WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

Using COBIT Indicators for Measuring

Scrum-based Software Development
VILJAN MAHNIC, NATASA ZABKAR
Faculty of Computer and Information Science
University of Ljubljana
Trzaska 25, SI-1000 Ljubljana
SLOVENIA
[email protected], [email protected], http://ltpo.fri.uni-lj.si/

Abstract: - The aim of this paper is to determine the level of compliance of AGIT model, developed during our
previous research for measuring Scrum-based software development, with the information systems auditing
criteria. For this purpose we use COBIT model. After a short introduction of Scrum, AGIT and COBIT, we
perform comparison analysis of their indicators for software development. Then we upgrade AGIT model with
the selected COBIT indicators. In order to improve the clarity of the model, we present its structure using IT
Balanced Scorecard. Finally we suggest possible further research.

Key-Words: - Scrum, Agile software development, IT performance measurement, IT indicators, IT Balanced

Scorecard, COBIT, AGIT

1 Introduction prescribed by software quality models including the

Software development projects are often of key need for comprehensive metrics plans. Many
importance for the achievement of the authors have recognized and explored a need for
organisation’s mission and objectives in the more elaborate metrics for agile development ([1],
effective and efficient, transparent and auditable [27], [4]).
manner [7]. There have been many attempts to Our previous research in this area is summarized
reduce the risks of software defects and improve the in the AGIT (AGIle software developmenT) model,
quality of software development process. These which includes basic indicators for measurement of
attempts include introduction of agile methods and Scrum-based software development [15], the
recently introduction of project management introduction of CMMI Measurement and Analysis
mutation model, which incorporates agile methods Practices into Scrum-based Software Development
and all other methods developed so far [20]. Process [17] and a description of corresponding
Introduction of agile methods can change and measurement repository [16]. In this paper we
improve project management practices [3], decrease further explore indicators of AGIT model. Our
overtime and increase customer satisfaction [19]. intention is to determine the level of compliance
The success rate is 41% for the agile projects and with the information systems auditing criteria. For
16% for the waterfall projects, according to [31], this purpose we use COBIT (Control Objectives for
who refers to The Standish Group 2006 research Information and Related Technology) [8], the IT
report. governance framework that is generally accepted in
XP and Scrum are the most commonly used agile the information systems auditing community and
methods [24]. In the last few years several commonly used for IT governance implementation
successful implementations of Scrum have been and assessment. COBIT has also been used from the
reported in the literature ([22], [30], [28], [2]). auditing perspective of agile development in [5], for
According to [28], the usage of Scrum reduces determining compliance of the projects using agile
every category of work (defects, rework, total work techniques with Sarbanes Oxley Act (SOX), a
required, and process overhead) by almost 50%, regulatory requirement for all public listed
when used in CMMI level 5 compliant company. companies in United States.
In this paper we focus on Scrum performance After a short introduction of Scrum, AGIT and
measurement. Like many other agile software COBIT model, we describe COBIT indicators for
development methods, Scrum follows the principle system development life cycle. Then we explain our
of “maximizing the amount of work that need not be compliance criteria. After that we compare Scrum,
done”. Therefore, it abandons many practices AGIT and COBIT indicators and discuss the results

ISSN: 1109-2750 1605 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

of this comparison. This is followed by the proposal Scrum teams often use the Scrum board for
of the adjustments to our model, including new tracking [Sut07b, Gup08]. On Scrum boards they
presentation of its structure. In the end, we give post Scrum burndown chart, prioritized list of
conclusions and plans for future work. impediments, columns of user stories, development
tasks relating to each story and the state of each
development task and tests associated with each
2 Scrum story, which can be further visualized by lamps that
change color depending on the state of the build or
2.1 Introduction to Scrum state of the Sprint. The usage of Scrum board
In this paper we assume that the reader is already eliminates the need for most status reporting, since
familiar with Scrum-based software development managers can see the state of the team in a few
process [23]. seconds. This is even easier if critical information is
One of the key Scrum tools for the performance put online on a web page, a wiki, or a reporting tool.
measurement is a direct day-to-day monitoring at According to [1], Scrum provides four simple
the 15-minute Daily Scrum meeting. At this meeting and effective artifacts for managing and monitoring
every Team member answers three questions: project performance:
• What have you done on this project since • Product Backlog,
the last Daily Scrum meeting? • Product Burndown,
• What will you do before the next • Sprint Backlog and
meeting? • Sprint Burndown.
• Do you have any obstacles? The progress can be presented in a general view like
ScrumMaster is responsible for these meetings and a burndown, which is easy to follow and understand
for resolving impediments encountered during the by managers, and a detail level like a backlog,
Sprint in order to assure smooth running of the which is useful for tracking the results.
development process. During these meetings data In CMMI level 5 compliant companies [10],
for the performance measurement can be gathered. Scrum progress (of sprints) is primarily measured
through the sprint burn down chart and the sprint
2.2 Scrum Indicators review meeting. [28] suggests combination of
Within Scrum originally only one software Scrum Burndown Chart and defect data, since
development metric is used. This original Scrum Scrum compensates other indicators by its high
indicator is the estimate of the amount of work level of interaction at the meetings
remaining that needs to be done in order to complete The indicators stated above are based on the
a Product Backlog item or a task in a Sprint original Scrum indicator: estimate of the amount of
Backlog. work remaining. In the next section we introduce
The total amount of work remaining is usually additional indicators, without violating the
shown in the burndown chart [23]. An example of principles of agility.
the burndown chart is presented in Fig. 1, based on
the data from [13].
3 AGIT Model
300

250 3.1 Introduction to AGIT Model

200 Experience has shown ([13], [14]) that, beside
150 estimating the amount of work remaining, it is
100 useful to measure the information about the amount
50 of work spent. This can be easily collected during
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
daily Scrum meetings and used for calculating the
Expected Burndown
Earned Value [21].
Based on this experience, we have developed
AGIT (AGIle software development) model.
Fig. 1: Sprint burndown Chart According to the principles of stakeholder driven
process performance measurement [12], the best
We can see that Scrum Team was on the schedule performance is achieved when the goals of all
until the 5th day, when they started going behind the stakeholders are satisfied. This requires a balanced
schedule (the “burndown” line is above “expected” approach considering viewpoints of different
line), but was back on the course at the 9th day.

ISSN: 1109-2750 1606 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

stakeholders, so AGIT model describes the stakeholder, appropriate performance indicators and
appropriate indicators for each stakeholder [15]. metrics that enable the evaluation of each indicator,
In AGIT model the views of four different taking into account that indicators should describe
stakeholders are considered: the process quantitatively and qualitatively [15].
• IT management, AGIT also includes a generic data model of
• Team members, measurement repository for collecting and storing
• ScrumMaster and measurement results [16].
• Customers. The important care was taken not to violate the
The first stakeholder is IT management who is principles of agility. All metrics (except the number
mainly concerned with traditional aspects of of errors reported by the user after release) can be
software development performance considering collected during meetings already prescribed by
time, cost, and quality. The second stakeholder is Scrum, thus not requiring a substantial additional
Team members whose main goal is “Job effort of the Team.
satisfaction”. The ScrumMaster is the third
stakeholder with the main goal of “Efficient 3.2 AGIT Indicators
impediments resolution”. Finally, the main goal The AGIT indicators for the previously stated goals
regarding the customers, the fourth stakeholder, is are shown in Table 1. Detailed description and
“Customer Satisfaction”. formulas of these 12 indicators can be found in
The top-down approach has been used in AGIT ([15], [17]).
model, in order to define the goals of each

Table 1: Indicators for Scrum-based Software Development Process (AGIT)

Stakeholder Goal AGIT Indicator
AG1: Timely AG1-1: Work Effectiveness
IT Information on
(ratio between the work spent and the decrement of work remaining)
Management Project AG1-2: Schedule Performance Index (SPI)
Performance (ratio between the earned value (i.e., the value of all tasks completed) and the
planned value (i.e., the initial estimate of value of all tasks to be completed till a
certain point within the project))
AG1-3: Cost Performance Index of Labor Costs (CPI)
(ratio between the earned value (measured in units of currency) and actual costs)
Quality AG1-4: Error Density
Improvement (number of errors per KLOC (kilo-lines of code))
AG1-5: Costs of Rework
(product of hours spent on rework and cost of an engineering hour)
AG1-6: Fulfilment of Scope
(ratio between the number of tasks completed in the Sprint and total number of
tasks in the Sprint Backlog or between the number of PBIs completed in the
release and total number of PBIs committed)
AG2: Job Satisfaction AG2-1: The Average Amount of Overtime at Sprint/Release/Project level
Team Members (the expected hours, the amount of work spent and administrative days)
AG2-2: The Average Number of Projects the Employees Work in Parallel
AG2-3: Qualitative Evaluation of Working Conditions
(communication and teamwork, physical discomfort, psychological well-being,
workload, supervision, opportunities for growth, etc.)
AG3: Efficient AG3-1: Average Number of Impediments per Task/Sprint/Team
ScrumMaster Impediments AG3-2: Mean Time for Resolving an Impediment (at Task/Sprint/Team level)
Resolution
AG4: Customer AG4-1: Qualitative Evaluation of Customer Satisfaction
Customers Satisfaction (the quality of product, price adequacy, reliability in terms of time and costs,
completeness of product delivered at the end of each Sprint or release, flexible
handling of changes in requirements, good collaboration with the development
team, adequate training and documentation, etc)
AG1-4: Error Density
AG1-6: Fulfilment of Scope

ISSN: 1109-2750 1607 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

4 COBIT Model These characteristics are mainly compliant with the

agility principles that value individuals and
4.1 Introduction to COBIT Model interactions over processes and tools, working
COBIT (Control Objectives for Information and software over comprehensive documentation,
Related Technology) [8] represents a collection of customer collaboration over contract negotiation,
documents which can be classified as generally and responding to change over following a plan
accepted best practice for IT governance, control [18].
and assurance. Our hypothesis is, that by satisfying The COBIT indicators can be used as key
information systems auditing criteria, we can performance indicators (KPI), key goal indicators
demonstrate that Scrum indicators proposed by (KGI) or key risk indicators (KRI). They can be
AGIT are (or are not) compliant with good practice. integrated in different models, such as Corporate IT
The IT processes are usually ordered into the Risk Management model [25].
responsibility domains of plan, build, run and In this paper we assess the compliance of AGIT
monitor. Within the COBIT framework, these and COBIT indicators and explore whether non-
domains are called: compliant or partly-compliant COBIT indicators can
• Plan and Organise (PO): Provides be integrated in AGIT model.
direction to solution delivery (AI) and
service delivery (DS); 4.2 COBIT Indicators for Software
• Acquire and Implement (AI): Provides the Development Process
solutions and passes them to be turned into In order to assess Scrum-based development process
services; indicators, we need to select the indicators for those
• Deliver and Support (DS): Receives the COBIT processes that relate to software
solutions and makes them usable for end development. There are few possible options ([5],
users; [6], [7], [32]).
• Monitor and Evaluate (ME): Monitors all Our selection of COBIT processes (and its
processes to ensure that the direction indicators) is based on:
provided is followed. • ISACA IS Auditing Guideline for System
Across these four domains, shown in Fig. 2, COBIT Development Life Cycle (SDLC) Reviews
has identified 34 IT processes. (Guideline G23) [6],
• four AI processes (AI1, AI2, AI6, AI7)
selected by [5] and
Plan and Organise • additional two processes, which we have
considered to be relevant for the agile
development (PO7, DS10).
Acquire and Deliver and The guideline G23 states that COBIT guidance
Implement Support for the following 5 processes should be considered
relevant when performing the audit of the system
development life cycle:
Monitor and Evaluate
• Process PO8: “Manage Quality” is focused
on ongoing performance monitoring against
Fig. 2: COBIT Domains [8] predefined objectives.
• Process PO10: “Manage Projects” is
For each of its 34 processes COBIT defines goals focused on monitoring of project risks and
and metrics to define and measure their outcome progress.
and performance, based on the principles of the • Process AI1: “Identify Automated
balanced business scorecard (BSC), introduced by Solutions” is focused on identifying
Robert Kaplan and David Norton [11]. COBIT technically feasible and cost-effective
metrics have been developed with the following solutions.
characteristics in mind:
• Process AI2: “Acquire and Maintain
• A high insight-to-effort ratio; Application Software” is focused on
• Comparable internally an externally; ensuring that there is a timely and cost-
• Better to have a few good metrics; effective development process.
• Easy to measure. • Process DS5: “Ensure Systems Security” is
focused on defining IT security policies,

ISSN: 1109-2750 1608 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

plans and procedures, and monitoring, applications and infrastructure solutions are
detecting, reporting and resolving security fit for the intended purpose and free from
vulnerabilities and incidents. errors, and planning releases to production.
According to [5], the following phases of system Finally, we have slightly broadened this selection
development life cycle can be mapped to agile by adding two COBIT processes which are, in our
development: opinion, important for achieving the goals of
• Understanding Requirements, stakeholders Team Members (PO7) and
• Designing Solutions, ScrumMaster (DS10):
• Building Solutions, • Process PO7: “Manage Human Resources” is
• Testing Solutions and focused on hiring and training personnel,
• Implementing Solutions. motivating through clear career paths,
These phases are mapped to four AI processes (AI1, assigning roles that correspond with skills,
AI2, AI6, AI7). The first two processes are already establishing a defined review process,
included in the G23 selection so we added the creating position descriptions and ensuring
following two processes: awareness of dependency on individuals.
• Process AI6: “Manage Changes” is focused • Process DS10: “Manage Problems” is
on controlling impact assessment, focused on recording, tracking and
authorisation and implementation of all resolving operational problems;
changes to the IT infrastructure, investigating the root cause of all significant
applications and technical solutions; problems; and defining solutions for
minimising errors due to incomplete request identified operations problems.
specifications; and halting implementation The 26 indicators, stated in the process
of unauthorised changes. description for these 9 COBIT processes, are
• Process AI7: “Install and accredit Solutions presented in Table 2.
and Changes” is focused on testing that

Table 2: COBIT Indicators for System Development Life Cycle

COBIT Process COBIT Indicators
Domain PO
PO7: Manage PO7-1: Level of stakeholders’ satisfaction with IT personnel expertise and skills
Human Resources PO7-2: IT personnel turnover
PO7-3: Percent of IT personnel certified according to job needs
PO8: Manage PO8-1: Percent of stakeholders satisfied with IT quality (weighted by importance)
Quality
PO8-2: Percent of IT processes that are formally reviewed by QA on a periodic basis and that
meet target quality goals and objectives
PO8-3: Percent of processes receiving QA review
PO10: Manage PO10-1: Percent of projects meeting stakeholders expectations (on time, on budget and meeting
Projects requirements—weighted by importance)

PO10-2: Percent of projects receiving post-implementation reviews

PO10-3: Percent of projects following project management standards and practices
Domain AI
AI1: Identify AI1-1: Number of projects where stated benefits were not achieved due to incorrect feasibility
Automated Solutions assumptions
AI1-2: Percent of feasibility studies signed off by the business process owner
AI1-3: Percent of users satisfied with functionality delivered
AI2: Acquire and AI2-1: Number of production problems per application causing visible downtime
Maintain Percent of users satisfied with the functionality delivered (AI2-2=AI1-3)
Application
Software
AI6: Manage AI6-1: Number of disruptions or data errors caused by inaccurate specifications or incomplete
changes impact assessment
AI6-2: Amount of application or infrastructure rework caused by inadequate change
specifications
AI6-3: Percent of changes that follow formal change control processes

ISSN: 1109-2750 1609 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

AI7: Install and AI7-1: Amount of application downtime or number of data fixes caused by inadequate testing
Accredit Solutions AI7-2: Percent of systems that meet expected benefits as measured by the post-implementation
and Changes process
AI7-3: Percent of projects with a documented and approved testing plan
Domain DS
DS5: Ensure DS5-1: Number of incidents damaging the organisation’s reputation with the public
Systems Security
DS5-2: Number of systems where security requirements are not met
DS5-3: Number of violations in segregation of duties
DS10: Manage DS10-1: Number of recurring problems with an impact on the business
Problems
DS10-2: Percent of problems resolved within the required time period
DS10-3: Frequency of reports or updates to an ongoing problem, based on the problem severity

5 Compliance Assessment AGIT indicator AG4-1: “Qualitative Evaluation of

Customer Satisfaction using Criteria” is compliant
with this COBIT indicator.
5.1 Method of Comparison
PO7-2: IT personnel turnover
The compliance assessment is performed by
Status: Partly compliant
determining the level of compliance for each COBIT
This COBIT indicator is partly covered by the three
indicator, selected in the previous section.
AGIT indicators that relate to stakeholder “Team
We compare the amount of information provided
members “:
by each COBIT indicator with the information
provided by selected AGIT indicators or Scrum • AG2-1: The Average Amount of Overtime
method. The possible results of comparison are: at Sprint/Release/Project level;
• Compliant, • AG2-2: The Average Number of Projects
the Employees Work in Parallel;
• Partly compliant and
• AG2-3: Qualitative Evaluation of Working
• Non-compliant.
Conditions.
If the information is comparable, then the indicators
PO7-3: Percent of IT personnel certified
are compliant. If the amounts of information are
according to job needs
significantly different, but still comparable, we mark
indicators as partly compliant. Finally, if the Status: Non-compliant
There is no AGIT indicator that would be compliant
information needed for COBIT indicator cannot be
with this COBIT indicator.
provided through any of AGIT indicators, we put
PO8-1: Percent of stakeholders satisfied with IT
these COBIT indicators in the non-compliant
quality (weighted by importance)
category.
Status: Compliant
Since COBIT addresses higher level of software
This COBIT indicator can be mapped to AGIT
development process than AGIT, the comparison of
indicator AG4-1: “Qualitative Evaluation of
these indicators is not always possible at the
indicator level. This is in line with the results of Customer Satisfaction using Criteria”. Apart from
this, at the end of the Sprint, every stakeholder can
Scrum and CMMI comparison ([28], [10]), where
assess product quality at a Sprint review meeting at
many of the generic practices were considered to be
which the Team presents what was developed
integrated in Scrum and therefore 100% compliant.
during the Sprint to the Product Owner and any
other stakeholders who want to attend.
5.2 Comparison Results PO8-2: Percent of IT processes that are formally
The results of comparison are presented in tables for reviewed by QA on a periodic basis and that meet
each domain (Table 3-5). target quality goals and objectives, PO8-3: Percent
of processes receiving QA review
5.2.1 Domain Plan and Organise (PO) Status: Compliant
The results for domain PO are shown in Table 3. These two COBIT indicators are covered by the
PO7-1: Level of stakeholders’ satisfaction with Sprint retrospective meeting. According to [10],
IT personnel expertise and skills Scrum teams can use a quality assurance schedule
Status: Compliant (QAS), where it is outlined what quality activities

ISSN: 1109-2750 1610 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

will be used to ensure the quality objectives are there could be different views about the level of
achieved. compliancy. However, the primary goal of post-
PO10-1: Percent of projects meeting implementation review is to assess the effectiveness
stakeholder’s expectations (on time, on budget and and efficiency of the IT solutions and their
meeting requirements—weighted by importance) implementation, initiate actions to improve the
Status: Compliant solution (where necessary) and serve as a learning
This COBIT indicator can be mapped to the tool for the future. This is reached through Scrum
following three AGIT indicators: Sprint Retrospective meeting, which supports the
• AG1-1: Work Effectiveness; goal of learning across projects by collecting the
• AG1-2: Schedule Performance Index (SPI); results from individual projects [28] and is an
• AG1-3: Cost Performance Index of Labor opportunity for the team to discuss what’s working
Costs (CPI). and what’s not working, and agree on changes to try
PO10-2: Percent of projects receiving post- [5]. Therefore, our final decision was that this
implementation reviews COBIT indicator is partly compliant.
Status: Partly compliant PO10-3: Percent of projects following project
Since Scrum Retrospective meeting happens management standards and practices
immediately after Scrum Review meeting and Status: Compliant
before the next Scrum Planning meeting, and the This COBIT indicator is covered by the
post-implementation review is scheduled at a ScrumMaster role, under the assumption that Scrum
reasonable time after the IT solution has been is used as a project management standard.
implemented (four weeks to six months) (G29, [6]),

Table 3: COBIT Indicators for System Development Life Cycle – Domain PO

COBIT Indicator Status AGIT Indicator/Scrum requirement
PO7-1: Level of stakeholders’ satisfaction with IT Compliant AG4-1: Qualitative evaluation of customer
personnel expertise and skills satisfaction
PO7-2: IT personnel turnover Partly AG2-1: The average amount of overtime
compliant AG2-2: The average number of projects the
employees work in parallel
AG2-3: Qualitative evaluation of working
conditions
PO7-3: Percent of IT personnel certified according Non-compliant
to job needs
PO8-1: Percent of stakeholders satisfied with IT Compliant AG4-1: Qualitative evaluation of customer
quality satisfaction
Scrum: Participation at the Sprint review
meeting
PO8-2: Percent of IT processes that are formally Compliant Scrum: Sprint retrospective meeting
reviewed by QA on a periodic basis and that meet
target quality goals and objectives
PO8-3: Percent of processes receiving QA review Compliant Scrum: Sprint retrospective meeting

PO10-1: Percent of projects meeting stakeholders Compliant AG1-1: Work effectiveness

expectations AG1-2: Schedule Performance Index
AG1-3: Cost Performance Index of Labor
Costs
PO10-2: Percent of projects receiving post- Partly Scrum: Sprint retrospective meeting
implementation reviews compliant
PO10-3: Percent of projects following project Compliant Scrum: Covered by the ScrumMaster role
management standards and practices assuring that Scrum is used as a project
management standard

5.2.2 Domain Acquire and Implement (AI) AI1-1: Number of projects where stated benefits
The results for domain AI are shown in Table 4. were not achieved due to incorrect feasibility
assumptions
Status: Partly compliant

ISSN: 1109-2750 1611 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

This COBIT indicator is partly covered by the AGIT In this case the rework is due to inaccurate
indicator AG1-6: “Fulfillment of Scope”, under the specifications or incomplete impact assessment.
assumption that the reasons for non-completion of AI6-2: Amount of application or infrastructure
the Sprint tasks are related to incorrect feasibility rework caused by inadequate change specifications
assumptions. Status: Compliant
AI1-2: Percent of feasibility studies signed off This COBIT indicator is indirectly covered by the
on by the business process owner following two AGIT indicators:
Status: Compliant • AG1-4: Error Density;
This COBIT indicator is covered by the Sprint • AG1-5: Costs of Rework.
planning meeting. In this case the rework is due to inadequate change
AI1-3, AI2-2: Percent of users satisfied with specifications.
functionality delivered AI6-3: Percent of changes that follow formal
Status: Compliant change control processes
These two COBIT indicators as well as previously Status: Compliant
described COBIT indicator PO8-1 (Percent of This COBIT indicator is covered by the Daily Scrum
stakeholders satisfied with IT quality) can be meeting, when changes are managed according to
mapped to AGIT indicator AG4-1: “Qualitative Scrum.
Evaluation of Customer Satisfaction using Criteria”. AI7-1: Amount of application downtime or
AI2-1: Number of production problems per number of data fixes caused by inadequate testing
application causing visible downtime Status: Compliant
Status: Compliant This COBIT indicator is indirectly covered by the
This COBIT indicator is indirectly covered by the following two AGIT indicators:
following two AGIT indicators: • AG1-4: Error Density;
• AG1-4: Error Density; • AG1-5: Costs of Rework.
• AG1-5: Costs of Rework. In this case the rework is due to inadequate testing.
AGIT uses the number of errors reported by the user AI7-2: Percent of systems that meet expected
in a fixed period after release as well as benefits as measured by the post-implementation
classification of tasks in the Sprint Backlog process
according to the type of work performed Status: Compliant
(development, testing, rework due to the change in This COBIT indicator can be mapped to AGIT
requirements, rework due to error reported by the indicator AG4-1: “Qualitative Evaluation of
customer, etc.). Customer Satisfaction using Criteria”.
AI6-1: Number of disruptions or data errors AI7-3: Percent of projects with a documented
caused by inaccurate specifications or incomplete and approved testing plan
impact assessment Status: Compliant
Status: Compliant This COBIT indicator is covered by the Sprint
This COBIT indicator is indirectly covered by the Backlog, under the condition that the tasks in the
following two AGIT indicators: Sprint Backlog include testing of components. This
• AG1-4: Error Density; way AI7-3 can be covered by AGIT without
• AG1-5: Costs of Rework. violating the principles of agility.

Table 4: COBIT Indicators for System Development Life Cycle – Domain AI

COBIT Indicator Status AGIT Indicator/Scrum requirement
AI1-1: Number of projects where stated benefits were Partly AG1-6: Fulfilment of scope
not achieved due to incorrect feasibility assumptions compliant

AI-2: Percent of feasibility studies signed off on by the Compliant Scrum: Sprint planning meeting
business process owner
AI1-3, AI2-2: Percent of users satisfied with Compliant AG4-1: Qualitative evaluation of
functionality delivered customer satisfaction
AI2-1: Number of production problems per application Compliant AG1-4: Error density
causing visible downtime AG1-5: Costs of rework
AI6-1: Number of disruptions or data errors caused by Compliant AG1-4: Error density
inaccurate specifications or incomplete impact AG1-5: Costs of rework
assessment

ISSN: 1109-2750 1612 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

AI6-2: Amount of application or infrastructure rework Compliant AG1-4: Error density

caused by inadequate change specifications AG1-5: Costs of rework
AI6-3: Percent of changes that follow formal change Compliant Scrum: Daily Scrum meeting
control processes
AI7-1: Amount of application downtime or number of Compliant AG1-4: Error density
data fixes caused by inadequate testing AG1-5: Costs of rework
AI7-2: Percent of systems that meet expected benefits Compliant AG4-1: Qualitative evaluation of
as measured by the post-implementation process customer satisfaction
AI7-3: Percent of projects with a documented and Compliant Scrum: Sprint Backlog
approved testing plan

5.2.3 Domain Deliver and Support (DS) AGIT indicator AG1-4: “Qualitative Evaluation of
The results for domain DS are shown in Table 5. Customer Satisfaction using Criteria” is compliant
DS5-1: Number of incidents damaging the with this COBIT indicator.
organisation’s reputation with the public, DS5-2: DS10-2: Percent of problems resolved within the
Number of systems where security requirements are required time period
not met, DS5-3: Number of violations in Status: Compliant
segregation of duties This COBIT indicator is covered by the following
Status: Compliant two AGIT indicators:
AGIT indicator AG1-4: “Qualitative Evaluation of • AG3-1: Average Number of Impediments
Customer Satisfaction using Criteria” is compliant per Task/Sprint/Team;
with these COBIT indicators; under the condition • AG3-2: Mean Time for Resolving an
that questionnaire includes questions regarding Impediment (at Task/Sprint/Team level).
security requirements that are result of development DS10-3: Frequency of reports or updates to an
activities. ongoing problem, based on the problem severity
DS10-1: Number of recurring problems with an Status: Compliant
impact on the business This COBIT indicator is covered by the
Status: Compliant ScrumMaster role, so the frequency is daily.

Table 5: COBIT Indicators for System Development Life Cycle – Domain DS

COBIT Indicator Status AGIT Indicator/Scrum requirement
DS5-1: Number of incidents damaging the organisation’s Compliant AG4-1: Qualitative evaluation of
reputation in public customer satisfaction

DS5-2: Number of systems where security requirements Compliant AG4-1: Qualitative evaluation of
are not met customer satisfaction

DS5-3: Number of violations in segregation of duties Compliant AG4-1: Qualitative evaluation of

customer satisfaction
DS10-1: Number of recurring problems with an impact on Compliant AG4-1: Qualitative evaluation of
the business customer satisfaction

DS10-2: Percent of problems resolved within the required Compliant AG3-1: Average number of impediments
time period AG3-2: Mean time for resolving an
impediment
DS10-3: Frequency of reports or updates to an ongoing Compliant Scrum: Daily Scrum meeting
problem, based on the problem severity

5.3 Interpretation of the Comparison AI7-2, AI7-3, DS5-1, DS5-2, DS5-3, DS10-1,
Results DS10-2, DS10-3).
The results, presented in Tables 2-5 show, that Only 3 of 26 (11%) applicable COBIT indicators
the majority or 22 of 26 (85%) applicable COBIT can be partly mapped to AGIT indicators (PO7-2,
indicators are covered by the AGIT indicators (PO7- PO10-2, AI1-1).
1, PO8-1, PO8-2, PO8-3, PO10-1, PO10-3, AI1-2, Finally, only 1 of 26 (4%) applicable COBIT
AI1-3, AI2-1, AI2-2, AI6-1, AI6-2, AI6-3, AI7-1, indicators is not included in AGIT model (PO7-3).

ISSN: 1109-2750 1613 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

Since 85% of applicable COBIT indicators are For the stakeholder “Team Members” we have
compliant with AGIT indicators, we can say that the added the following two indicators:
AGIT model almost completely satisfies COBIT • PO7-2: “IT personnel turnover” and
criteria. • PO7-3:“Percent of IT personnel certified to job
Non-compliant COBIT indicator does not depend needs”.
on the software development method (in our case For the stakeholder “IT Management” we have
Scrum), but is related to the human resources added the following two indicators:
strategy (PO7-3: Percent of IT personnel certified to • PO10-2: “Percent of projects receiving post-
job needs). implementation reviews” and
The same applies to the partly compliant • AI1-1: “Number of projects where stated
indicators, that relate to PO7-2: IT personnel benefits were not achieved due to incorrect
turnover (partly covered by AG2-1: The average feasibility assumptions”.
amount of overtime, AG2-2: The average number of
projects the employees work in parallel and AG2-3:
Qualitative evaluation of working conditions),
6.2 IT Balanced Scorecard
After adjusting the indicators of the AGIT model,
PO10-2: Percent of projects receiving post-
we wanted to adjust the presentation of its structure
implementation reviews (partly covered by Sprint
as well. In order to increase its clarity and
retrospective meeting) and AI1-1: Number of
familiarity to the executive management, we have
projects where stated benefits were not achieved due
decided to use the balanced scorecard form of
to incorrect feasibility assumptions (partly covered
presentation [11].
by AG1-6: Fulfilment of scope).
The primary goal of BSC is to transform strategy
into action and allow management to monitor the
6 Further AGIT Development implementation of the strategy. BSC includes
performance measurement from four perspectives
6.1 Adjusting AGIT Indicators [9], as shown in Fig. 3.
In this section we introduce our solution for non-
compliant and partly compliant indicators. Financial
Non-compliant COBIT indicator PO7-3: “Percent Perspective
of IT personnel certified to job needs” can be
calculated by keeping records of the team members’ Internal
Customer Business
certificates. We have decided to add this indicator to Perspective Process
AGIT model. Perspective
Partly compliant COBIT indicator PO7-2: “IT
personnel turnover” can be calculated by keeping Learning and
Growth
records not only on the number of all developers, as Perspective
recommended in AGIT model, but also on the
number of the developers that left the company and
the number of the new developers (in the certain Fig. 3: BSC (adjusted from [11])
period of time). Partly compliant COBIT indicator
PO10-2: “Percent of projects receiving post- IT has its own balanced scorecard (IT BSC), with
implementation reviews” (partly covered by Sprint redefined four perspectives ([8],[26]):
retrospective meeting) can be calculated by keeping • Enterprise contribution: How do business
records abut post-implementation reviews. The last executives view the IT department?
partly compliant COBIT indicator AI1-1: “Number • User orientation: How do users view the IT
of projects where stated benefits were not achieved department?
due to incorrect feasibility assumptions” can be • Operational excellence: How effective and
calculated by introducing classification of the efficient are the IT processes?
causes of the non-completion of the tasks for the • Future orientation: How well is IT
indicator “fulfilment of scope” (one of the possible positioned to meet future needs?
causes could be incorrect feasibility assumptions). BSC and IT BSC (Fig. 4) enables reader to capture
We have decided to add these three indicators to all four perspectives at once. We find that this
AGIT model. presentation is clear and simple, and we prefer this
The final result was the addition of the four new approach to the sequential table approach used so
COBIT indicators to AGIT model. far. Therefore we have decided to harmonize the

ISSN: 1109-2750 1614 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

presentation of adjusted AGIT model structure with •

AGIT-0.1: the first published version of the
the IT BSC presentation form. model in which the key indicators were
presented [15];
• AGIT-0.2: the second published version of
Enterprise the model that included presentation of the
Contribution
data repository design [16].
One of our criteria for good measurement system
User Information Operational is simplicity and flexibility. In order to increase
Orientation Excellence
simplicity of our model we have mapped our four
stakeholders to the four perspectives of IT BSC:
Future • IT Management - Enterprise contribution;
Orientation
• Customer - User orientation;
Fig. 4: IT BSC (adjusted from [8]) • ScrumMaster - Operational excellence;
• TeamMembers - Future orientation.
6.3 Solution for AGIT-0.3 The adjusted indicators and adjusted presentation
For the purpose of this paper we have given the can be seen in the Table 6. This way, each
adjusted AGIT model name AGIT-0.3, assuming stakeholder can monitor the implementation of its
that the previous two versions are: goals in easy and simple way, in compliance with
Agility Principles.

Table 6: AGIT-0.3 Indicators for Scrum-based Software Development Process

AG1: IT Management
AG1-1: Work Effectiveness
Performance
Information
on Project
Timely

AG1-2: Schedule Performance Index

(SPI)
AG1-3: Cost Performance Index of
Labor Costs (CPI)
AG1-4: Error Density
AG1-5: Costs of Rework
AG1-6: Fulfilment of Scope
Improvement

PO10-2: Percent of projects receiving

post-implementation review
Quality

AI1-1: Number of projects where

stated benefits were not achieved due
to incorrect feasibility assumptions
AG4: Customers AG3: ScrumMaster
300
AG4-1: Qualitative 250
AG3-1: Average
Evaluation of Number of Impediments
Impediments

200
Satisfaction

Resolution

Customer per Task/Sprint/Team

Customer

Efficient

150

Satisfaction 100

50
AG1-4: AG3-2: Mean Time for
0
Error Density 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Resolving an
AG1-6: Fulfilment of Expected Burndown
Impediment (at
Scope Task/Sprint/Team level)
AG2: Team Members
AG2-1: The average amount of overtime
Job Satisfaction

AG2-2: The average number of projects the

Employees work in parallel
AG2-3: Qualitative evaluation of working
conditions
PO7-2: IT personnel turnover
PO7-3: Percent of IT personnel certified
according to job needs

ISSN: 1109-2750 1615 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

7 Conclusion Hawaii International Conference on System

Measurement of agile software development has Sciences (HICSS07), pp. 1-9
been explored by many authors ([1], [27], [4]). Our <http://csdl2.computer.org/comp/proceedings/
previous research in this area is summarized in the hicss/2008/3075/00/30750466.pdf>
AGIT (AGIle software developmenT) model, [3] M. Ceschi et al., Project Management in Plan-
which includes basic indicators for measurement of Based and Agile Companies, IEEE Software,
Scrum-based software development [15], CMMI May/June 2005, pp. 21-27.
Measurement and Analysis Practices for Scrum- [4] E. Damiani, A. Colombo, F. Frati, C.
based Software Development Process [17] and a Bellettini, A Metamodel for Modeling and
description of corresponding measurement Measuring Scrum Development Process,
repository [16]. Lecture Notes in Computer Science 4536,
The aim of this paper was to assess AGIT model Springer-Verlag Berlin Heidelberg, 2007, pp.
by determining the level of compliance with the 74-83.
information systems auditing criteria, as described [5] S. Gupta: SOX Compliant Agile Processes,
in COBIT (Control Objectives for Information and Proceedings of AGILE 2008 Conference
Related Technology) [8], the IT governance (AGILE’08), pp. 140-143.
framework commonly used for IT governance [6] ISACA, IS Standards, Guidelines and
implementation and assessment. Procedures for Auditing and Control
After short introduction of the Scrum, AGIT and Professionals, Information Systems Audit and
COBIT models, we compared the appropriate Control Association, Rolling Meadows, 2008.
indicators from COBIT model for system [7] ITAF, A Professional Practices Framework
development life cycle with the indicators proposed for IT Assurance, Information Systems Audit
in AGIT model. For this purpose we used the and Control Association, Rolling Meadows,
selection of COBIT processes as defined in the 2008.
information systems auditing guideline G23 [6], [8] ITGI, COBIT v4.1, Information Technology
slightly extended by selection for agile Governance Institute, Rolling Meadows, 2007.
development by [5] and additional two processes [9] ITGI, IT Governance Implementation Guide:
that, in our opinion, refer to agility. We compared Using COBIT and ValIT, 2nd Edition,
26 indicators for the selected COBIT processes with Information Technology Governance Institute,
12 indicators from AGIT model. The results of Rolling Meadows, 2007.
comparison were one non-compliance and three [10] C. R. Jakobsen, K. A. Johnson, Mature Agile
partial-compliances, which did not depend on the with a twist of CMMI, Proceedings of AGILE
software development method. Therefore we 2008 Conference (AGILE’08), pp. 212-217.
concluded that model AGIT almost completely [11] R. S. Kaplan, D. P. Norton, The Balanced
satisfies COBIT criteria. Then we introduced our Scorecard: translating strategy into action,
solution for these non/partly compliances in the President and Fellows of Harvard College,
new version of the model, named AGIT-0.3. In the 1996.
end we simplified the presentation of AGIT-0.3 [12] P. Kueng, Process performance measurement
model structure using BSC cross form. system: a tool to support process-based
We hope that the results of this paper contribute organizations, Total Quality Management,
to the quality and clarity of AGIT model. In the Vol. 11, No. 1, 2000, pp. 67-85.
near future we plan to map this model to other [13] V. Mahnic, S. Drnovscek, Agile Software
comparable and accepted models. Project Management with Scrum, EUNIS 2005
Conference – Session papers and tutorial
References: abstracts, University of Manchester, June
[1] B. Barton, K. Schwaber, D. Rawsthorne, 2005, CD-ROM.
Reporting Scrum Project Progress to <http://www.mc.manchester.ac.uk/eunis20
Executive Management through Metrics 05/medialibrary/papers/paper_194.pdf>
(2005, January) http://www.scrumalliance.org/ [14] V. Mahnic, S. Drnovscek, Introducing agile
system/resource/file/21/Reporting_Scrum_Pro methods in the development of university
ject_Progress_to_Executive_Management_thr information systems. Proceedings of the 12th
ough_Metrics.pdf > International Conference of European
[2] B. Barton, E. Campbell, Implementing a University Information Systems EUNIS 2006,
Professional Services Organization Using Tartu, June 2006 (2006), pp. 61-68.
Type C Scrum, Proceedings of 40th Annual

ISSN: 1109-2750 1616 Issue 10, Volume 7, October 2008

 WSEAS TRANSACTIONS on COMPUTERS Viljan Mahnic, Natasa Zabkar

[15] V. Mahnic, I. Vrana, Using stakeholder driven Development, IEEE Software, May/June 2005,
process performance measurement for pp. 36-42.
monitoring the performance of a Scrum based [23] K. Schwaber, Agile Project Management with
software development process, Scrum, Microsoft Press, 2004.
Electrotechnical Review, Ljubljana, Vol. 74, [24] C. Schwaber et al., The Truth About Agile
No. 5, 2007, pp. 241-247. Processes (2007, December 3).
[16] V. Mahnic, N. Zabkar, Measurement <www.forrester.com>
repository for Scrum-based software [25] M. Spremic, M. Popovic, Emerging issues in
development process, Modern topics of IT Governance: implementing the corporate IT
computer science : proceedings of the 2nd risks management model, WSEAS
WSEAS International Conference on Transactions on Systems, Issue 3, Volume 7,
Computer Engineering and Applications March 2008, pp. 219-228.
(CEA'08), Acapulco, Mexico, January 25-27, [26] M. Spremic, Z. Zmirak, K. Kraljevic,
2008 pp. 23-28. Evolving IT Governance Model – Research
[17] V. Mahnic, N. Zabkar, Introducing CMMI Study on Croatian Large Companies, WSEAS
Measurement and Analysis Practices into Transactions on Business and Economics,
Scrum-based Software Development Process, Issue 5, Volume 5, May 2008, pp. 244-253.
International Journal of Mathematics and [27] T. Sulaiman, B. Barton, T. Blackburn,
Computers in Simulation, Issue 1, Volume 1, AgileEVM - Earned Value Management in
2007, pp. 65-72. Scrum Projects, Proceedings of AGILE 2006
[18] Manifesto for Agile Software Development, Conference (AGILE’06), pp. 7-16.
2001 <http://www.agilemanifesto.org/> [28] J. Sutherland et al., Scrum and CMMI Level 5:
[19] C. Mann, F. Maurer, A Case Study on the The Magic Potion for Code Warriors,
Impact of Scrum on Overtime and Customer Proceedings of AGILE 2007, pp. 272-278
Satisfaction, Proceedings of the Agile [29] J. Sutherland, K. Schwaber, The Scrum
Development Conference (ADC’05), pp. 70- Papers: Nuts, Bolts, and Origins of an Agile
79. Process,
[20] E. Markopoulos, J. Bilbao, E. Bravo, T. Draft 10/14/2007
Stoilov, T. E. J. Vos, C. Figa´Talamanca, K. <www.jeffsutherland.com/scrum/ScrumPapers
Reschwamm, Project Management Stage .pdf>
Mutations within Agile Methodological [30] B. Upender, Staying Agile in Government
Framework Process Transformations, WSEAS Software Projects, Proceedings of the Agile
TRANSACTIONS on INFORMATION Development Conference (ADC’05), pp. 153-
SCIENCE & APPLICATIONS, Issue 5, 159.
Volume 5, 2008, pp. 776-785. [31] T. Wailgum, From Here to Agility (2007,
[21] PMI, A Guide to Project Management Body of December 3).
Knowledge, third edition (PMBOK Guide), < http://www.cio.com>
Project Management Institute, Inc., Four [32] N. Zabkar, V. Mahnic, Using COBIT Model in
Campus Boulevard, Newtown Square, Software Development, Proceedings of
Pensylvania, 2004. Slovenian Informatics Conference 2005 (DSI
[22] B. Schatz, I. Abdelshafi, Primavera Gets 2005), April 13-15, 2005, Portoroz, Slovenia,
Agile: A Successful Transition to Agile pp. 257-262

ISSN: 1109-2750 1617 Issue 10, Volume 7, October 2008