A Guide to Monitoring and Evaluation of

Capacity-Building Interventions in the Health

Sector in Developing Countries

March 2003
MEASURE Evaluation Manual Series, No. 7

MEASURE Evaluation Project

Anne LaFond, MS, JSI Research and Training
Lisanne Brown, PhD, Tulane University

The manual series is made possible by support from USAID under the terms of Cooperative
Agreement HRN-A-00-97-00018-00. The opinions expressed are those of the authors, and do
not necessarily reflect the views of USAID.

March 2003

Printed on recycled paper

Other Titles in the Manual Series

NO. 1

Evaluando Proyectos de Prevencin de VIH/SIDA: Un Manual con Enfoque

en las Organizaciones No Gubernamentales. July 2000.

NO. 2

Quick Investigation of Quality (QIQ): A User's Guide for Monitoring Quality of Care. February 2001.

NO. 3

Sampling Manual for Facility Surveys for Population, Maternal Health,

Child Health and STD Programs in Developing Countries. July 2001.

NO. 4

Measuring Maternal Mortality from a Census: Guidelines for Potential Users, July 2001.

NO. 5

A Trainer's Guide to the Fundamentals of Monitoring and Evaluation for

Population, Health, and Nutrition Programs, 2002.

NO. 6

Compendium of Indicators for Evaluating Reproductive Health Programs,

August 2002.

Recommended Citation

LaFond, Anne and Brown, Lisanne. A Guide to Monitoring and Evaluation of Capacity-Building Interventions in the Health Sector in Developing Countries. MEASURE Evaluation Manual Series, No. 7.
Carolina Population Center, University of North Carolina at Chapel Hill. 2003.

Acknowledgements
We wish to acknowledge the contributions and support of a number of individuals and institutions that enabled the successful completion of this document. Ray Kirkland and Krista Stewart
of USAID were instrumental in the conception of the Guide. Sara Pacque-Margolis of USAID
provided the support to see it through to completion. Our sincere gratitude also goes to several
technical reviewers for their constructive and instructive comments on earlier versions of the
Guide. They are: Alfredo Fort (PRIME II), Diane Catotti (IPAS), Alison Ellis (MSH), Leo Ryan
(CSTS/ORC Macro), Eric Sarriot (CSTS/ORC Macro), Fred Carden (IDRC), and Doug Horton
(ISNAR). Kate Macintyre contributed her ideas and encouragement, as well as provided the
SAIDIA case material. Catherine Elkins and Kate Macintyre contributed to the MEASURE
working paper on measuring capacity in the health sector, which provided a basis for this guide.
Thom Eisele and Cira Endley reviewed and analyzed capacity-measurement tools and practices.
Case examples of capacity measurement were developed with the cooperation of PRIME /
INTRAH; SAIDIA; NGO Networks for Health; and PATH (in a workshop setting). Finally, we
are grateful to the many adventurous organizations and individuals working to build capacity in
the health sector in developing countries. Their experimentation in capacity-building monitoring
and evaluation is commendable and deserves further study. This guide would not have been possible without the support of the Offices of Health and Population at the United States Agency for
International Development (Contract Number: HRN-A-00-97-00018- 00).

Acknowledgements

Prologue
Capacity development1 has moved to center stage of the agendas of development organizations.
Substantial sums are being invested in capacity-building programs. Yet, their design and management leave much to be desired. Marred by untested, unrealistic assumptions, the results of
many programs fall short of their goals and expectations.
Evaluations are needed to test the theories and assumptions on which capacity development
programs are based, to document their results, and to draw lessons for improving future programs. However, few capacity development programs have been systematically and thoroughly
evaluated (Horton et al., 2000).

Capacity building and capacity development are used interchangeably throughout this document.

Prologue

iii

List of Acronyms and Abbreviations

API
BASICS
CHW
DHS
DOSA
FHI
FP
FPE
FPEI
FPMD
FPPE
HR
IAI
IDA
IDRC
IEC
IHFA
IISD
ISNAR
M&E
MDA
MEASURE
MES
MFSS
MIS
MOH
MOST
MSH
NGO
OCAT
OSI
PASCA
PHR
PI
PROSE
PSI
RH
SAIDIA
SFPS
STD
TOT
WHO

AIDS Program Effort Index

Basic Support for Institutionalizing Child Survival
Community Health Worker
Demographic and Health Survey
Discussion-Oriented Organization Self-Assessment
Family Health International
Family Planning
Family Planning Effort
Family Planning Effort Index
Family Planning Management Development Project
Family Planning Program Effort
Human Resources
Institutional Assessment Instrument
Institutional Development Assessment
International Development Research Centre
information, education, and communication
Integrated Health Facility Assessment
International Institute for Sustainable Development
International Service for National Agricultural Research
Monitoring and Evaluation
Management Development Assessment
Monitoring and Evaluation to ASsess and Use REsults
Materials, Equipment, and Supplies
Management/Financial Sustainability Scale
Management Information System
Ministry of Health
Management and Organizational Sustainability Tool
Management Sciences for Health
Nongovernmental Organization
Organizational Capacity Assessment Tool
Outcome Sustainability Index
Program for NGOs that provide HIV/AIDS services in Central America
Partnership for Health Reform
performance improvement
Participatory, Results-Oriented Self-Evaluation
Program Sustainability Index
Reproductive Health
Local Kenyan NGO
Sant Familiale et Prvention du SIDA
Sexually Transmitted Disease
Training of Trainers
World Health Organization

Acronyms and Abbreviations

Table of Contents
Acknowledgements.......................................................................................................................... i
Prologue ......................................................................................................................................... iii
List of Acronyms and Abbreviations.............................................................................................. v
About This Guide............................................................................................................................ 1
Structure of the Guide................................................................................................................. 2
Introduction..................................................................................................................................... 3
Defining Capacity-Building Monitoring and Evaluation ........................................................... 4
Capacity-Building M&E Has Many Roles ................................................................................. 5
Part 1. Concepts, Definitions, and Attributes of Capacity and Capacity Building ........................ 7
Why Build Capacity?.................................................................................................................. 7
What is Capacity Building? ........................................................................................................ 7
Useful Definitions ...................................................................................................................... 7
Attributes of Capacity and Capacity Building............................................................................ 7
Capacity Building Is Behavior Change ...................................................................................... 9
Why Monitor and Evaluate Capacity Building?....................................................................... 11
What Is Different about M&E of Capacity Building?.............................................................. 11
Implications for Capacity-Building M&E ................................................................................ 12
Summary for Managers and Evaluators.................................................................................... 12
Part 2. Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual
Framework ........................................................................................................................ 15
Overview Framework: The Role of Capacity in the Health Sector.......................................... 15
Capacity at a Single Level ........................................................................................................ 17
Defining Variables Related to Capacity and Performance ....................................................... 18
Using These Conceptual Frameworks ...................................................................................... 25
Summary for Managers and Evaluators.................................................................................... 26
Part 3. Monitoring and Evaluating Capacity-Building Interventions........................................... 27
STEP 1 Define the Purpose of the Evaluation......................................................................... 28
STEP 2 Define Performance Objectives.................................................................................. 30
Defining Performance........................................................................................................... 30
STEP 3 Mapping Capacity: Build a Conceptual Framework for a Specific Capacity-Building
Intervention ................................................................................................................ 32
When to Map Capacity ......................................................................................................... 32
How to Map Capacity ........................................................................................................... 33
Single-Level Capacity Mapping ........................................................................................... 34
Multi-Level Capacity Mapping............................................................................................. 36
Dealing with Context ............................................................................................................ 36
Interpreting and Using Capacity Maps ................................................................................. 41
Table of Contents

vii

STEP 4 Identify Capacity Indicators ....................................................................................... 44

What Are Capacity Indicators?............................................................................................. 44
Working with Capacity Indicators ........................................................................................ 45
Lessons for Indicator Development ...................................................................................... 45
STEP 5 Identify Appropriate Methodological Approach and Sources of Data....................... 55
Methodological Approaches and Challenges........................................................................ 55
Tackling Methodological Challenges ................................................................................... 56
Sources of Data ..................................................................................................................... 57
Tools for Measuring Capacity at Different Levels ............................................................... 58
STEP 6 Develop an Implementation and Dissemination Plan ................................................ 65
Part 4. Summary Checklist: Steps for Designing a Capacity-Building M&E Plan ..................... 67
Checklist: Steps in Designing a Capacity-Building M&E Plan ............................................... 67
Annex A. Example of Scoring Used for Measuring Capacity Building in Training, PRIME I . 71
Annex B. Example of Results of PRIME Training Capacity Index .......................................... 75
Annex C. Key Internet Resources for Monitoring and Evaluating Capacity-Building
Interventions ............................................................................................................... 77
Annex D. Capacity Mapping and Performance Improvement Compared .................................. 91
Glossary ........................................................................................................................................ 93
Bibliography ................................................................................................................................. 95

viii

Tables
Table 1

The Use of Assessment vs. M&E in Capacity-Building Intervention ........................ 4

Table 2

Capacity and Performance Variables Defined .......................................................... 20

Table 3

Questions Posed by Different Types of Capacity-Building M&E ............................ 28

Table 4

Examples of Capacity Indicators in Current Use in Health Programs ...................... 52

Table 5

Examples of Performance Indicators in Current Use in Health Programs................ 53

Table 6

Example of a Table of Data Sources for an Organizational Assessment .................. 60

Table 7

Capacity Measurement Tools .................................................................................... 61

Figures
Figure 1

Overview of Capacity in the Health Sector............................................................... 16

Figure 2

Health System Capacity ............................................................................................ 19

Figure 3

Health Service and Civil Society Organization Capacity ......................................... 21

Figure 4

Health Program Personnel Capacity.......................................................................... 23

Figure 5

Individual/Community Capacity .............................................................................. 24

Boxes
Box 1

Capacity Measurement Case Examples ...................................................................... 1

Box 2

Measuring the Effectiveness of Capacity Building in Training: PRIME I ................. 8

Box 3

Examples of Organizational Capacities .................................................................... 10

Box 4

Six Steps for Developing a Capacity-Building M&E Plan ....................................... 27

Box 5

DO'S AND DONTS of Developing an M&E Plan for a Capacity-Building

Intervention ............................................................................................................... 29

Box 6

Characteristics of a Good Performance Objective .................................................... 31

Box 7

The Process of Capacity Mapping............................................................................. 32

Tables, Figures and Boxes

ix

Box 8

Questions to Guide Discussion for Capacity Mapping ............................................. 34

Box 9

Guidance on Capacity Mapping ................................................................................ 35

Box 10

Questions to Guide Discussion on the External Environment and Its Influence on

Organizational Capacity ............................................................................................ 42

Box 11

Examples of Capacity Indicators from Non-health Sector Capacity-Building

Interventions.............................................................................................................. 46

Box 12

PASCA: From Self-Assessment to External Assessment ......................................... 56

Box 13

Advantages and Disadvantages of Self-Assessment and External Assessment

Techniques................................................................................................................. 59

Maps
Map 1

Organizational Capacity Map - Single Level ............................................................ 37

Map 2

Organizational Capacity Map - Single Level ............................................................ 38

Map 3

Organizational Capacity Map - Single Level ............................................................ 39

Map 4

Community Capacity Map on Multiple Levels......................................................... 40

Map 5a

Mapping Capacity First Iteration .............................................................................. 48

Map 5b

Mapping Capacity Second Iteration .......................................................................... 49

Map 6

Community Capacity Map on Multiple Levels with Indicators................................ 50

About This Guide

This guide has grown out of the collective
experience of health and development organizations working to build health sector capacity in developing countries. The focus of the
Guide is the measurement of capacity for the
purpose of monitoring and evaluating capacity-building interventions. It responds to a
demand among public health planners, evaluators, and practitioners for advice on assessing the many aspects of health programming
that fall under the rubric of capacity building.
The purpose of this guide is to assist health
planners and evaluators to

gain a clear understanding of the concepts

of capacity and capacity building

critically evaluate the strengths and limitations of current approaches to capacity
measurement
design a capacity-building M&E plan that
outlines a systematic approach to measuring capacity and assessing the results of
capacity-building interventions in the
health sector
The Guide was developed based on a request
from the United States Agency for International Development under the MEASURE
Evaluation Project.
Many readers of this guide may not be aware
that there is a lack of widespread experience
in the field of capacity-building M&E in the
health sector. Capacity-building programs
proliferate. Yet, methods for testing and
tracking their results are rare. We have therefore based the advice in this guide mainly on
lessons learned from current practices in capacity assessment (see Table 1 for discussion
of the differences between assessment and
M&E). Sources include: a review of the state
of the art of capacity measurement (Brown,

About This Guide

LaFond, and Macintyre, 2001); a review of

capacity-building measurement tools and indicators; formal and informal consultations
with practitioners; and an in-depth exploration
of four different capacity measurement experiences (Box 1). The Guide also draws on
lessons learned about capacity-building
monitoring and evaluation in other sectors,
such as agriculture and housing, and on new
evaluation approaches designed to support
learning in development programming (Horton et al., 2000; Morgan, 1997; Earl, Carden,
and Smutylo, 2001).
Box 1: Capacity Measurement Case
Examples

SAIDIA, a health and community development nongovernmental organization

(NGO) in Kenya;
the PRIME I and PRIME II index of
capacity of training institutions;
A Workshop on Sustainability and Capacity Building hosted by PLAN International in May 2001, in Dakar, Senegal;
and
MEASURE Program Technical Assistance to NGO Networks for Health.

From the discussion that follows on the concept of capacity building and capacity measurement techniques readers will come to understand why this guide is neither prescriptive
nor exhaustive. Standardized approaches to
monitoring and evaluating capacity-building
interventions are not found because of the
wide variety of circumstances in which capacity building takes place. Capacity building
has been applied to actions as distinct as policy formulation, supplying basic health commodities, and identifying danger signs of

malnutrition. In short, capacity building demands adaptation to its context and capacitybuilding evaluation techniques must reflect
this potential variation. The Guide acknowledges this and other challenges by providing a
link between the theoretical and practical aspects of capacity measurement in the health
sector and offering an approach to monitoring
and evaluation that is relevant in a variety of
settings.
It is also important to keep in mind that the
monitoring and evaluation of capacity building, while singled out for discussion in this
document, is normally part of an overall plan
or system for monitoring and evaluating a
health program or health sector intervention.
This guide should therefore be used as a tool
for orienting planners to capacity measurement in the context of developing a projectlevel or overall program-level performancemonitoring plan (particularly programs where
sustainability and scaling-up are a central
concern). As such, it will aid the process of
thinking through the role capacity and capacity measurement play in improving performance.

Structure of the Guide

Part 1 of the Guide briefly discusses attributes of capacity and capacity building, and
how these attributes influence M&E approaches.
Part 2 introduces a series of conceptual
frameworks for understanding the role of capacity in the health sector and illustrates possible capacity variables (&) at each level of
the health system.

The heart of the Guide is found in Part 3,

which suggests a 6-step approach to developing an M&E Plan for Capacity-building
that centers on the process of capacity mapping (&). Mapping involves the construction
of a visual framework that helps the evaluator
understand relationships (or assumed relationships) among the many factors that contribute
to or detract from capacity and, ultimately,
performance. Mapping can be used to identify
untapped, constrained, or missing elements of
capacity. It also can be used to guide intervention choices and to build a monitoring and
evaluation framework. Part 3 also comments
on indicator selection for M&E and practical
lessons from field experience, as well as
methods and data sources, and dissemination
of results. The indicators and tools referenced
in this section are provided as examples to
stimulate thinking and discussion about capacity-building and M&E strategies rather
than as prescribed approaches.
Part 4 concludes the Guide with a summary
and checklist for developing a capacitybuilding M&E plan. Annexes contain details
of M&E approaches and a summary of Webbased resources on capacity-building M&E.
The Glossary at the end of the Guide explains
many of the technical words and jargon used
in the field of monitoring and evaluation. In
the text they are marked with the following
symbol: &

Introduction
Over the last decade, capacity building has
become as central to the business of developing health systems in lesser-developed countries as providing financial resources and applying the latest science. Capacity is believed
to contribute directly to improving performance in the health sector, and is thought to
play an important role in sustaining adequate
performance over time. Despite increased
attention to capacity, experience in gauging
the effectiveness of capacity-building interventions in the health sector is still limited.
Unlike other aspects of health-related monitoring and evaluation (M&E), capacity measurement is not supported by a comprehensive
history of theory and practice. While methods
for monitoring and evaluating health service
coverage, access, and quality are well advanced, there are few tried and true approaches for capturing the interim state or
process that reflects the ability to achieve and
sustain coverage, access, and quality over
time (Brown, LaFond, and Macintyre, 2001).
Thus, capacity measurement in the health
sector is both new and experimental.
There are intrinsic challenges to measuring
capacity that are reflected in the concept and
role of capacity itself. For example, capacity
derives its relevance from the contribution it
makes to performance. There are endless areas where performance is required in the
health sector, and an equally wide range of
possible capacity variables that influence performance. In addition, contextual factors (or
factors outside the control of most health
sector actors &) can have a strong influence
on capacity or the desired outcome of capacity-building intervention. These and other
characteristics of capacity and capacity
building explain why there are no gold standards for capacity-building M&E. There is no
short list of valid indicators of capacity in the
health sector, nor are there standardized
Introduction

measurement tools applicable to every capacity-building experience.

Many of these challenges have also discouraged widespread testing of methods of capacity-building monitoring and evaluation. The
extent of experience is so limited that, at this
stage, capacity measurement is considered to
be an art rather than a science. Evaluators
must therefore approach M&E of capacitybuilding interventions with a willingness to
test strategies and share what they have
learned in order to build a body of theory and
practice.
Despite the conceptual and practical challenges of tackling capacity measurement,
there are a number of reasons to put energy
and time into developing a sound approach to
monitoring and evaluation of capacitybuilding interventions. The most significant
reason is that measurement is an important
part of achieving capacity-building and performance goals. Monitoring and evaluation
can help health program professionals understand the relationship between capacitybuilding interventions, capacity and performance, and to focus strategies used for improving performance. Specifically, monitoring and evaluation can help answer a range of
questions about
the process of capacity change (how capacity building takes place),
capacity as an intermediate step toward
performance (what elements of capacity
are needed to ensure adequate performance), and
capacity as an outcome (whether capacity
building has improved capacity)

Table 1: The Use of Assessment vs. M&E in Capacity-Building Intervention

Capacity Assessment

Capacity Monitoring and Evaluation

q

Purpose: diagnostic or descriptive; defines

constraints
Measures gap between actual and desired
performance
Findings are used for internal purposes
(design and planning)
One time measurement
Action oriented

Looks broadly at existing situation

q
q
q
q

In this guide, when we talk about monitoring

and evaluation of capacity building or
capacity development, we are mainly interested in the last question, that is, measuring
changes in capacity and linking them (directly
or indirectly) to capacity-building interventions.

Defining Capacity-Building
Monitoring and Evaluation
Most capacity measurement experience to
date has emphasized capacity assessment
rather than M&E (Brown, LaFond, and Macintyre, 2001). Assessment normally takes
place at the beginning of an intervention as
part of an organizational diagnosis or formative design process. Evaluators can learn a
great deal from capacity assessment tools (as
we have in developing this guide). However,
it is worth noting that while capacity assessment is an important first step in planning a
capacity-building intervention, capacitybuilding M&E differs from assessment by
virtue of its explicit focus on measuring
change. Capacity-building monitoring and
evaluation tracks or identifies changes in capacity that take place in the course of a capacity-building intervention. It uses stated
objectives for capacity building and perform-

q
q
q
q

Purpose: predictive; for accountability or

comparisons; gauges results
Measures results or progress toward desired results
Findings are used for internal and external
purposes (management; accountability)
Often uses repeat measurement
Action, analysis and accountability oriented
Uses conceptual frameworks to discern
relationships between variables
ance improvement as a reference for gauging
progress. As such, it guides program management as well as informs funding agencies
about the results of capacity-building investments. A final aspect of M&E (as opposed to
diagnosis or assessment) is the use of conceptual frameworks that make assumptions
about the relationship between different variables that influence capacity and performance. Table 1 describes many of the differences between capacity assessment and M&E.

In the evaluation of capacity development, the impact metaphor should

be avoided. The militaristic impact
metaphor fails to capture the essential features of capacity development,
which is a process of change and
growth. (Horton, 2002).

Capacity monitoring normally would be used

to understand the effectiveness and efficiency
of a capacity-building intervention during
implementation (i.e., is capacity improving
and at what cost?), to contribute to strategic or
operational decisions related to capacity
building, or to enable a periodic look at a program or system. Capacity evaluation is normally more complex than monitoring, and is

conducted to gain understanding of the relationship between capacity-building interventions and capacity outcomes, or the links between capacity and performance variables.
The term impact evaluation & is not appropriate or useful in the context of capacitybuilding M&E because of the difficulty of
quantifying many elements of capacity and
attributing capacity change to any single intervention or even a range of interventions.

Capacity-Building M&E Has Many

Roles
A final introductory observation relates to the
role that measurement plays in a capacitybuilding intervention. Many experienced capacity-building practitioners feel that capacity
measurement cannot be separated from the

Introduction

process of building capacity itself. Program

managers often use capacity assessment tools
to raise awareness about capacity problems,
stimulate commitment to improving capacity
among stakeholders, and for setting selfdetermined benchmarks. The focus is internal.
In practice, capacity-building M&E is often
encouraged (or required) by external
stakeholders to be used mainly for accountability. Defining the purpose of M&E is
therefore not always easy for managers and
evaluators. The discussion that follows considers the pros and cons of these various approaches and informs critical measurement
choices. It begins with a discussion of the
rationale for capacity-building M&E and explores the concept of capacity and its role in
improving performance.

Part 1

Concepts, Definitions, and Attributes of Capacity and

Capacity Building

Why Build Capacity?

In the context of results-based programming,
resources are invested in different aspects of
the health sector with the ultimate aim of enhancing health system performance and improving the health of populations. Translating
these resources into sustained performance
often requires new or improved capabilities in
individuals and organizations (including
communities) operating in the health sector.
Capacity represents the potential for using
resources effectively and maintaining gains in
performance with gradually reduced levels of
external support.

What is Capacity Building?

Used alone, the term capacity building is intangible and vague. What constitutes capacity
building in practice can vary enormously, and
the concept continues to develop as field experience grows. In the early days of capacitybuilding intervention, many practitioners
equated capacity building with training. If
there was a gap in performance, the solution
was often to hold a workshop to retrain or
refocus the individuals whose performance
was faltering. Organizational development
experts and field-level capacity-building efforts inform us today that individual skills are
only part of the complex mixture of elements
that constitute capacity to perform a certain
function or groups of functions effectively
and consistently over time. Individual health
workers, no matter how skilled, are unlikely
to deliver essential health and family planning
services effectively without adequate supplies
and equipment, proper motivation and support, a good relationship with the community
served by the health center, and so on. Capacity building may be required in all of these

and other areas to ensure performance goals

are met.

Useful Definitions &

It is useful to start with definitions of capacity, capacity building and performance to
guide measurement efforts and M&E planning. This guide returns frequently to such
issues because meaningful capacity measurement depends on clear understanding of capacity and its role in the health sector.
Capacity is the ability to carry out stated
objectives (Goodman et al, 1998). It has also
been described as the stock of resources
available to an organization or system as well
as the actions that transform those resources
into performance (Moore, Brown, and Honan,
2001).
Capacity building (or capacity development)
is a process that improves the ability of a
person, group, organization, or system to
meet objectives or to perform better.2
Performance is a result or set of results that
represent productivity and competence related to an established objective, goal or
standard.

Attributes of Capacity and Capacity

Building
The definitions of capacity and capacity
building above reflect certain attributes of
each concept that inform this guides approach to monitoring and evaluation. These
attributes are as follows:
2

For other definitions of capacity building, see Brown,

LaFond and Macintyre, 2001.

Concepts, Definitions, and Attributes of Capacity and Capacity Building

Box 2: Measuring the Effectiveness of Capacity Building in Training:

PRIME I
The PRIME I project provided technical assistance to strengthen the capacity of
local institutions in developing countries to train health personnel for reproductive
health (RH) service delivery. A key M&E strategy for this project was development
of a capacity index specific to the features of RH training institutions. The index
was tested in 14 countries and later revised and applied to monitor the results of
PRIMEs capacity building in training activities. Detailed reports of these evaluations
in El Salvador, Dominican Republic, Ghana, and other countries are available from the
PRIME II project (Catotti, 1999; Ampomah, 2000; Luoma, 2000; www.prime2.org).

Capacity building can be defined only

in terms of a specific objective or goal.

In the health sector, capacity does not exist
for its own sake. Health planners and managers are concerned with capacity because it
enables performance. For example, a health
facility that experiences regular stock-outs of
pharmaceuticals might require additional capacity in financial planning or supplies management (i.e., interventions that are specific to
the particular performance goal of commodity
supply). It follows that a capacity development strategy for improving pharmaceutical
supply would call for a different approach
than one aimed at strengthening community
involvement in health. The link between capacity and performance, therefore, serves as
the guide for both programming and evaluation of capacity-building interventions. Improved performance, in turn, is a good indicator of success in capacity development.

Capacity (and capacity building) are

dynamic and volatile.

Capacity can be perceived as a moving target.
At any given time, capacity can improve or
decline. It often develops in stages that indicate improved readiness to influence performance (Goodman et al., 1998). Capacity building, therefore, is an ongoing process (the development of abilities), whose stages can be
measured as development outcomes &

through monitoring and evaluation. The dynamic nature of capacity is often a reflection
of the many different forces that influence its
development or decline.

Capacity building is multidimensional.

Capacity building can be described in terms

of levels. In the health sector, capacity is required at four different levels: health system,3
organization, health personnel, and community. Yet, to date, most capacity-building experience and measurement have focused on
organizational and health personnel capacity.
In practice, capacity at one level is often influenced by actions at other levels. A single
missing aspect of capacity rarely explains
performance failures. The PRIME project
(Box 2), for example, constructed an index of
the capacity of training institutions that included 13 critical elements, ranging from political support for training in reproductive
health to community involvement in training
(Fort, 1999).
Analysis of capacity levels through measurement encourages evaluators to think in terms
of complex, multifaceted systems. Connections and forces within a system are critical to
3

Some have labeled this level institutional development (Kotellos, 1998; INTRAC, 1998), while others
use the terms organization and institution interchangeably. To avoid confusion, we have adopted the
term system.

understanding constraints to capacity and how

to overcome them. Paying too much attention
to one part of the organization or system may
limit results at the overall organization or
system level and fail to improve overall performance (Morgan, 1997). Take the example
of delivering immunization services at the
organization level. The effectiveness of this
service depends on elements that go beyond
the capacity of the facility alone. The Cold
Chain & must function from the central level
to the facility to ensure vaccine viability.
Civil service norms, regulations, and salary
levels can influence health worker motivation
and acceptance of the value of immunization
among caregivers and encouragement from
community leaders can affect service utilization. If performance falters (i.e., coverage
declines), it may be the result of limited capacity at the facility or other levels. An
evaluation framework should consider all
these variables, although it may focus measurement efforts on a smaller number of them.

Capacity depends on the context.

Contextual factors or elements of the external

environment influence capacity directly and
indirectly. Contextual influences include cultural, social, economic, political, legal, and
environmental variables. The influence of
these factors may be crucial to the success of
capacity building, yet they are often difficult
to control or measure. For example, Sierra
Leones Ministry of Health (MOH) may have
the capacity to deliver childhood immunization services. However, frequent political instability in the country can challenge that capacity and reduce performance (e.g., immunization coverage) dramatically. Taking a more
general example, the stagnation and decline of
economic growth that occurred in Africa in
the 1980s severely undermined public sector
capacity to meet recurrent costs for salaries
and supply of basic health commodities. Even
well-established health systems, such as
Ghanas, were unable to withstand the decline

in health sector financing, and capacity

gradually eroded to a very low level (LaFond,
1995).

Capacity Building Is Behavior

Change
In addition to these attributes, current thinking
about capacity building reflects two ways of
capturing the changes expected as a result of
intervention. Traditional approaches to capacity building concentrate on the internal
functioning of organizations and systems
(structures, strategies, staff, and skills).
Morgan (1997), however, notes the necessity
of considering the macro aspect of capacity
building that relates to the behavior and operations of groups of organizations or individuals and their role in wider systems (such
as the role of public sector health systems,
ministries of health, or district-level health
units in rural health improvement). In general,
there is more experience working on and
measuring capacity at the micro level than at
the macro level.
Taking both a micro and macro look at capacity building suggests that capacity development goes beyond a simple technical intervention. It is to a great extent focused on inducing behavior change, a process that involves learning, moderating attitudes, and
possibly adopting new values at individual,
organization, and system levels. Therefore,
the focus of capacity-building interventions
and M&E must capture related conditions and
concepts such as motivation, culture, and
commitment, as well as changes in resource
availability, skill levels, and management
structure (Morgan, 1997; James, 2001). Examples of different types of organizational
capacities are found in Box 3.

Concepts, Definitions, and Attributes of Capacity and Capacity Building

Box 3: Examples of Organizational Capacities

Six General Areas of Capacity in the CSTS Institutional Sustainability Assessment
1. Strategic management practices
2. Organizational learning
3. Use and management of technical knowledge and skills
4. Financial resource management
5. Human resource management
6. Administrative infrastructure and procedures
Source: Sarriot, 2002a
Structural or technical
The organization has effective program recruitment, development, and retention of staff
that it can perform its critical functions adequately. It must have a basic set of competencies that can enable it to cope with its workload and environment.
The organization has a structure, technology, and set of procedures that enable staff to
carry out the critical functions.
The organization has the ability, resources, and autonomy to focus on a manageable set of
objectives over a reasonable period of time. Its goals are reasonably clear, accepted, and
achievable.
The organization can alter its structure and functioning by including new actors, new partnerships, decentralization, delegation, creation of new organization, downsizing, privatization, participation, devolution, and changing responsibilities for government.
Behavioral
The organization understands the implications of its experiences and can change its collective behavior in line with this understanding. It can learn and adapt. It has a self-renewing
capacity.
The organization can form productive relationships with outside groups and organizations as
part of a broader effort to achieve its objectives. It can manage these relationships for
both its own gain and that of its partners.
The organization has the ability to legitimize its existence. It must be able to persuade key
external stakeholders of the value of supporting its continued functioning. It has an identity
that is accepted internally and externally, and the loyalty of its clients, customers, and
stakeholders gives it protection and resources.
The organization has a culture, a set of values, and organizational motivation that values and
rewards performance.
The work community has a population of groups and organizations that is sufficient to carry
out the tasks and services needed to implement such critical functions as analysis, production, mediation, communication, networking, fund-raising, and so on.
Source: Morgan, 1997

10

Why Monitor and Evaluate Capacity

Building?
Given the nature of capacity development
the volatility of capacity, its many levels, and
links to performancesome authors describe
capacity building as a high-risk investment
(UNICEF, 1999). Yet, most development
organizations agree that facilitating growth in
capacity among local partners systems, organizations, and communities is key to the
success of social development overall. As
such, all stakeholders need dependable methods for answering such questions as
What capacity exists now, and how does it
affect performance?
What improvements in capacity or new
kinds of capacity are required?
Is capacity being built? Is the capacitybuilding intervention focused on the right
elements?
What has been learned about capacitybuilding strategies?
How does capacity contribute to
sustainability?
In addition, there is value in not restricting
monitoring and evaluation of health and development interventions to a few important
outcomes or results (i.e., quality, coverage,
and health status). Organizations and systems
produce many different and critical effects.
For strategic purposes, and to manage change
in programs, organizations and systems effectively, regular information on a number of
operational indicators is required (Moore,
Brown and Honan, 2001). A well-defined
monitoring and evaluation strategy will help
make sense of these many facets of capacity
and performance. Monitoring and evaluation
should help local practitioners and their external partners to think strategically about
capacity development and to learn, through
practice, what works under different circumstances. At the same time, systematic measurement of capacity contributes to results-

based management of programs where capacity building is part of the overall strategy for
improving performance.

What Is Different about M&E of

Capacity Building?
Traditionally, monitoring and evaluation focuses more on measuring performance and
less on the way performance is achieved or
sustained. In contrast, capacity-building M&E
focuses fundamentally on processes (e.g.,
building alliances, mobilizing communities,
decentralized planning, learning) and other
qualitative aspects of individual or organizational change (e.g., motivation to perform)
that contribute to better performance. Consequently, M&E of capacity building often
seeks to capture actions or results that often
are not easily measured.
That said, results of capacity building are as
important as processes. In capacity-building
intervention, the process and result of capacity building becomes the intermediate outcome that is expected to lead eventually to
improved and sustained performance. Exploring the links between changes in capacity
and changes in performance is therefore key.
However, it often involves considerable
speculation about the capacity needed to
achieve those goals. One of the main gaps in
the knowledge base that informs capacity
measurement is the lack of common understanding of the relationship between capacity
and performance. Little is known about what
elements or combinations of elements of capacity are critical to performance. Moreover,
there is considerable variation in what constitutes adequate performance.

Concepts, Definitions, and Attributes of Capacity and Capacity Building

11

Implications for Capacity-Building

M&E
Clearly, the attributes of capacity and capacity
building noted above have implications for
monitoring and evaluation. Broadening the
concept of capacity building beyond technical
skills and resources and thinking about capacity building in terms of multiple levels and
influences helps planners and evaluators to
hypothesize about what aspects of capacity
are critical to performance and to define entry
points for targeting capacity-building interventions. A measurement approach should
also reflect a clear understanding of the interaction among different aspects of capacity
and how they work (or fail to) work together,
particularly with respect to individual and
organizational behavior. These types of variables may be represented by indicators in an
evaluation plan, but may require additional
interpretation to ensure a complete grasp of
capacity and its role in improving performance.
As noted in the Introduction, it is also important to keep in mind the conventional wisdom
about how to monitor and evaluate capacity.
Conventional wisdom notes that it is not productive to separate measurement practices
from capacity building itself (Morgan, 1997;
Horton, 2001; Earl et al, 2001). Because capacity-building M&E focuses on behavior
change, the success of capacity development
is often directly related to the extent of ownership and commitment to the process on the
part of the participants. This commitment
includes, in some cases, ownership of the
design, procedures, and reporting of monitoring and evaluation activities. Applied in
this way, monitoring and evaluation of capacity can become a key strategy for improving
performance. However, many of the M&E
methods that promote ownership (i.e., involve
self-evaluation and relying on respondents
perceptions) may also affect the validity of
findings. Specifically, they may compromise
12

the use of capacity-building M&E for accountability, predicting performance, or

making comparisons between different interventions or sites (common reasons for conducting evaluation). This theme surfaces often
in the discussion of capacity-building M&E,
and will be addressed in Part 3 of this guide.

Summary for Managers and

Evaluators

Capacity is a pre-condition for perform-

ance. Capacity building is used to improve

performance in a variety of ways and
situations.

Capacity-building M&E is normally part

of an overall plan or system for monitoring and evaluating a health program or

health sector intervention.

There are no standardized approaches for

capacity-building M&E because of the

wide variety of circumstances in which
capacity building takes place. There is no
short list of valid indicators of capacity in
the health sector, nor are there standardized measurement tools applicable to
every capacity-building experience.

Monitoring and evaluation should help

local practitioners and their external partners to think strategically about capacity
development and to learn, through practice, what works under different circumstances. At the same time, systematic
measurement of capacity contributes to
results-based management of programs
where capacity building is part of the
overall strategy for improving performance.

 Capacity building in the health sector can

be described and measured in terms of

four levels: health system, organization,
health personnel, and community. Capacity at one level can be influenced by actions at other levels.

Contextual factors or elements of the ex-

ternal environment influence capacity directly and indirectly.

ganizations. Thus, capacity-building M&E

must capture conditions and concepts such
as motivation, culture, and commitment,
as well as changes in resource availability,
skill levels, and management structure.

Any strategy monitoring capacity should

reflect a clear understanding of the interaction among different aspects of capacity

and how they work (or fail to work) together.

Capacity development goes beyond a

simple technical intervention, focusing on

behavior change in individuals and or-

Concepts, Definitions, and Attributes of Capacity and Capacity Building

13

Part 2

Understanding the Role of Capacity in the Health Sector:

Introducing a Conceptual Framework

The first step in developing a vision of capacity development, and a plan to measure it, is
to understand the role capacity plays in the
health sector in developing countries. What
are the expectations and assumptions surrounding capacity and its relationship to performance and health outcomes? Clear thinking about these variables helps planners define realistic objectives for capacity-building
interventions and express desired capacity
outcomes explicitly and precisely. Evaluators
must rely on these parameters of capacity
building in order to develop a capacitybuilding M&E plan.
The following series of conceptual frameworks are provided as a reference to help
planners and evaluators develop their own
vision of the role capacity (and capacity
building) plays in the health sector. We have
found that directed discussion using these
types of frameworks prior to M&E planning
can stimulate strategic thinking within project
or work teams, clarify individual and collective expectations and thereby improve capacity-building M&E. Figure 1 The Overview
illustrates the critical role capacity plays in
influencing and sustaining performance in the
health sector. It takes a system-wide view of
capacity, including all possible levels where
capacity building might take place. The four
other frameworks (Figures 2-5) take capacity
at each level and break it down into defined
components: inputs, processes, outputs, and
outcomes (See Table 2). In breaking down
capacity at each level, the frameworks provide a starting point for identifying the key
variables that influence capacity and performance at that level.

Overview Framework: The Role of

Capacity in the Health Sector
Health system performance depends on capacity. Figure 1 provides an overview of that
relationship and specifies four levels where
capacity is needed to ensure performance:
system, organization, health personnel, and
individual/community. The diagram suggests
that capacity contributes to performance at all
levels, and capacity at each level collectively
enables overall health system performance.
Understanding capacity and
performance of individuals and
organizations demands careful
consideration of their role in
larger systems, and their relationships within those systems
(Morgan, 1997).

Figure 1 also implies that capacity plays a

role in sustaining health system performance.
If health system performance remains adequate over time (supported by consistent capacity), performance is said to be sustained.
Although few health systems in developing
countries can boast this accomplishment, the
underlying aim of capacity development
should be a sustained change in resources or
behavior that leads to improved and sustained
performance. The goal is not short-term gain
but a lasting or robust change in ways of doing business that becomes imbedded in the
system or organization itself.

Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual Framework

15

Figure 1. Overview of Capacity in the Health Sector

External Environment

Capacity Levels

Performance

Health System

Health System
Performance

Sustainability
T
I

Organization
Health
Program
a
Personnel

Organizational
Performance
Personnel
Performance

M
E

Sustainable
Health
System
Performance

Improved
Health
Status

Individual/Community
Capacity

Individual/Community
Behavior Change
External Environment

16

Sustained
Individual/Community
Behavior Change

At the center of the framework is the ultimate

goal of capacity building in the health sector:
improved health status. Capacity does not
directly influence health status but contributes
to it through its link to performance at system,
organization and health personnel levels. In
this illustration, the health system interacts
with individuals or groups of individuals (e.g.,
the community) to influence health status.
Individuals and communities contribute to
health system capacity by interacting with
providers and organizations (receiving care,
determining priorities, or providing resources)
and to health system performance by using
health services. In addition, individuals and
communities can improve their health status
independent of the health system by promoting and adopting preventive measures, such as
regular hand washing, not smoking, or eating
well. Improvements in individual and community capacity should result in sustained
behavior change over time, representing this
levels contribution to sustained health system
performance and improved health status.
At the perimeter of Figure 1 we mark the influence of environmental or contextual factors, including cultural, social, economic,
political, legal, and environmental variables
that influence capacity and performance at all
four levels (Africa Bureau, 1999; Horton,
2001; James 2001). The obvious importance
of these factors for improving and sustaining
both capacity and performance suggests that
special efforts are needed for tracking their
status overtime. In this guide, we focus
mainly on variables that donors, governments,
private agencies, and individuals can influence through health sector interventions.
However, we also encourage evaluators to
identify and monitor key contextual variables
and examine their relationship to program
outcomes.

Capacity at a Single Level

The four levels of capacity are detailed further
in the following related frameworks (Figures
2-5).
These conceptual frameworks take a broad
look at capacity at one level to illustrate many
of the potential factors that might come together to influence capacity and performance.
The purpose of these frameworks is to show
how capacity can be broken down at each
level into inputs, processes, outputs, and outcomes in order to

identify the different factors that contrib-

ute to capacity, and performance

hypothesize about the potential relationships among these factors within a single
level
Conceptual frameworks like these differ from
logical or strategic frameworks in that they do
not reflect the linear logic of a particular capacity-building intervention, and its presumed
effect on capacity outcomes. Rather, they
show the range of all possible variables that
might influence capacity and performance. In
this way they help planners at the early design stages to determine the scope and focus
of a capacity-building intervention, and
evaluators to design valid measures for determining the success of those interventions.
Conceptual frameworks can become gradually more specific as decisions are made about
capacity-building interventions and the capacity and performance changes expected
from them.

Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual Framework

17

Defining Variables Related to

Capacity and Performance
Capacity inputs represent the resources (human, financial, material, etc.) that contribute
to capacity and performance. Processes represent the activities or behaviors at each capacity level that transform resources (inputs) into
capacity outputs and outcomes. Capacity outputs and outcomes are the results of inputs
and processes, and indicate products (outputs)
and an ability to carry out stated objectives
(outcomes). In many cases, capacity outcomes
are expressed as knowledge, skills and behavior. Performance is the expected result of
capacity (a stock of resources) and the environment, the final link in the hypothesized
chain of causality. Performance is defined as
results that represent productivity or competence related to an established objective, goal,
or standard.
System Level
Figure 2 refers to the health system. It includes the resources, actors, and institutions
related to the financing, regulation, and provision of health actions (Murray and Frenk,
1999; WHO 2000).4 The system is seen as a
collection of institutions or organizations,
plus the personnel in those organizations
working together to deliver health care and/or
promote better health. The health system performs certain functions independent of those
performed by the organizations, and personnel within it, and therefore possesses its own
capacity that can be assessed over time and
targeted for intervention.

tors of performance at this level.5 The framework includes a range of possible capacity
inputs, processes, outputs and outcomes that
contribute to performance at this level.
The system level is a complex area in which
to define or address capacity development or
to assess changes in capacity resulting from
external or internal intervention. Despite the
use of an inputs-process-outputs-outcomes
framework, in practice, relationships among
elements of capacity are not perfectly linear.
Change (or the lack of it) in capacity results
from multiple influences, some of which can
be unexpected (Sarriot, 2002a). Contextual
factors such as political and economic stability can also play a dominant yet poorly understood role in ensuring system capacity. Good
examples come from health sector reform
activities that seek to improve national health
sector performance by changing sector priorities, laws, organizational structures, and financing arrangements. For instance, the actual
results of legal reform in Zambia were
achieved but not well communicated to health
workers, which led to internal resistance to
delinking or separating health workers from
the civil service (Lake et al., 2000). Despite
addressing key constraints such as laws or
regulations, capacity to manage human resources more effectively did not emerge as
planned.

Performance at the health system level is often defined in terms of access to services,
quality of care, equity, and efficiency, although there are many other possible indica5

A health action is defined as any set of activities

whose primary intent is to improve or maintain health
(Murray and Frenk, 1999).

18

The World Health Organization proposed new indicators for monitoring health system performance in the
World Health Report 2000, including measures of
stewardship, financing, resource generation, and service provision.

Figure 2: Health System Capacity

External Environment

Inputs
Infrastructure
Public/private composition
of services
Organizational structure
(public sector)
Existing health-related
laws, regulations, and
policies
Information/
communication
systems
Human resources
Leadership
Financial resources (public/
private, internal/external)
History and culture of the
system

Outputs

Process
Health policy making

Effective health policies

Enforcement of health
related laws and
regulations

Published health policies

and regulations

Health sector strategic

planning

Formal and informal

coalitions

Resource allocation

Sector-wide strategy

Resource generation

Increased local financing

of recurrent costs

Financial management
Improved human resource
availability in rural areas

Human resource
development and
management

Coordinated donor
interventions

Donor coordination
Timely analysis and
dissemination of national
health information

Multi-sectoral
collaboration

Capacity
Outcomes

Information coordination
& dissemination

Accountability
(financial and program
transparency)
Capacity to assess and
cope with internal and
external change
Financial self-reliance
Effective monitoring of
quality of care
Responsiveness to client
needs and demands
Efficient/appropriate
resource allocation

P
e
r
f
o
r
m
a
n
c
e

Use of information for

strategy and learning

External Environment

Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual Framework

19

Table 2: Capacity and Performance Variables Defined

Input

Set of resources, including health personnel, financial resources, space, policy

orientation, and program service recipients, that are the raw materials that
contribute to capacity at each level (system, organization, health personnel,
and individual/community).

Process

Set of activities, practices, or functions by which the resources are used in

pursuit of the expected results.

Output

Set of products anticipated through the execution of practices, activities, or

functions.

Outcome

Set of results that represent capacity (an ability to carry out stated objectives),
often expected to change as a direct result of capacity-building intervention.

Performance

Set of results that represent productivity and competence related to an established objective, goal or standard. The four capacity levels together contribute
to overall system-level performance.

Impact

Long-term results achieved through improved performance of the health system: sustainable health system and improved health status. Impact measures
are not addressed in capacity-building M&E.

Organization Level
Figure 3 depicts a similar categorization of
capacity variables at the organization level
that contribute to organizational performance.
Performance at the organization level might
be described in terms of the ability of the organization to produce goods and services to
an acceptable standard (e.g., the quality of
care; coverage of the catchment population).
This framework relates to organizations
whose main function might be health service
delivery (in the public or private sector) and
those considered to be civil society organizations (nongovernmental or nonhealth service
agencies). Civil society organizations generally are not involved in the direct delivery of
health services, but they do influence health
service delivery, policies, and behaviors in
many societies throughout the world. Civil
society organizations of particular importance

20

could be cooperatives, community development organizations, advocacy groups, informal pressure groups, and others. The MOH is
a unique organization for conceptualizing
capacity building since it can be a significant
actor at both the system and organization levels. The contextual factors influencing organizational capacity are represented at the
perimeter of the diagram and include system
level factors as well as typical political, economic, cultural, and other variables.

Figure 3: Health Service and Civil Society Organization Capacity

External Environment

Inputs

Process

Infrastructure

Strategic and operational

planning

Organizational
structure

Human resource management

and development

Mission

Financial management

Leadership

Logistics/supplies
management

Financial
resources
Equipment and
Supplies
Human resources
(technical &
managerial)
History and
culture of
organization

Research and evaluation

Coordination with other units
Resource mobilization
IEC
Advocacy
Community relations and
mobilization

Outputs
Strategic and operational
plans

Capacity
Outcomes
Able to assess and cope with
internal and external
change

Staff trained and supported

Functional management
systems (i.e., supplies
available, supervision done)

Responsiveness to client
needs and demands
Financial self-reliance

Functional financial
management system (i.e.,
resources available, costs
contained)

Stakeholder involvement

Functional health
information and
communication system
(information collected,
analyzed and used)

Acting and learning with

information

Functional service delivery

systems (i.e., services
available)
Regular IEC and community
mobilization activities

Regular supply of essential

commodities/No stock outs

Ability to monitor service

quality and correct gaps as
needed

P
e
r
f
o
r
m
a
n
c
e

Able to develop and

maintain working
relationships with other
organizations and groups

External Environment

Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual Framework

21

Health Program Personnel Level

Figure 4 presents the health program personnel level. The term health personnel refers to
all those who perform clinical, managerial,
advocacy or other work within the health
system. In contrast to the system and organization levels, comprehensive interventions to
build and maintain capacity are more common at the health personnel level. Ideally, in
each health system there is a plan for producing and maintaining a cadre of qualified
personnel (personnel with capacity) and providing them with an adequately supportive
environment in which to perform effectively.
It is less common to find comprehensive organization- and system-level capacitybuilding plans, although one could argue they
are equally important.
The vast majority of capacity-building interventions in the health sector focus on changing the skills and behavior of health personnel
because managers and providers play a critical role in ensuring organization and system
level capacity and performance. This framework attempts to tease out some of the key
variables at this level that relate directly to
individual health personnel capacity, but we
must acknowledge that organizational context
is equally important. Organizations and systems are often responsible for the inputs and
processes that enable health personnel to perform effectively. Thus, there is a significant
overlap between the inputs and processes that
contribute to capacity at the organization and
the health personnel levels. Many of the variables listed in system and organization level
frameworks also contribute to health personnel capacity.

22

Inputs such as sufficient funds, space and

materials for professional development are
transformed into capacity outcomes through
processes such as educational and training
events or other opportunities for improving or
maintaining health personnel capacity. Capacity outcomes relate to the knowledge,
skills, experience, and motivation resulting
from inputs and processes. Performance at
this level includes the application of knowledge and skills in management, health services delivery, training, and other related activities.
Individual/Community Level
The final figure, Figure 5, represents the demand side of the equation for capacity
building as well as the role individuals and
communities play in shaping health systems
and improving health status. In addition to
system, organization, and health personnel
levels, capacity is required within individual
clients and communities to ensure demand for
appropriate services to promote their role in
contributing to or influencing service delivery, and to encourage the practice of certain
behaviors conducive to good health. For example, clients capacity to demand improved
or new services or to engage with health care
personnel and organizations is vital to health
system performance and achieving adequate
health status of the population.

Figure 4: Health Program Personnel Capacity

External Environment

Inputs

Process

Outputs

Capacity
Outcomes

Financial resources
(i.e., salaries, benefits,
incentives)
Physical resources
venues
materials
supplies
equipment
National/organizational
training policies, plans,
and guidelines
Up-to-date information
on appropriate clinical
and managerial
practices
Curricula

Pre-service and in-service

training events (training of
trainers and trainees)
Training events for
managers (including
supervisors)
Staff performance
evaluations
Experiential learning
opportunities
Professional networking

Staff trained/retrained
as required
Trainers
trained/retrained as
required

Knowledge and skills

of trainees

Managers
trained/retrained as
required

Trainers and trainees

continue to gain
experience

Supervision received

Motivated health
personnel

Professional or peer
support networks
Access to information

P
e
r
f
o
r
m
a
n
c
e

Human resources

External Environment

Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual Framework

23

Figure 5: Individual/Community Capacity

External Environment

Inputs
Individual/family
Education
Income
Family history
Sex
Perceptions of need/risk
Willingness to seek care
Ability to pay
Exposure to programs/services

Process

Needs identification and

problem solving
Collaboration

Past experiences with health

services and prevention
practices

Achieving consensus

Utilization-enhancing activities
(e.g., IEC, accessible services)

Securing resources

Critical reflection

Negotiation
Community dimensions
Community history
Communication
Citizen participation
Cohesiveness
Leadership
Material and financial resources
(internal and external to community)
Social and interorganizational
networks
Communication channels
Values
Skills

Outputs

Recognition of
need for services
Intention to use
services
Participation in
community health
committees
Community plans

External Environment

24

Capacity
Outcomes

Recognition of
symptoms and
danger signs and
actions needed
Ability to articulate
needs and demands
Knowledge of
prevention behavior
Community support
for prevention
behaviors
Community support
for communitybased health care
Community-based
mobilization and
empowerment for
interacting with
health system

P
e
r
f
o
r
m
a
n
c
e

Here the individual/community level represents all those who could benefit from and
participate in the health care system; thus it
includes all current and potential clients of the
services offered and the communities in
which they live. The inclusion of individual
and community capacity in this framework
represents a departure from conventional
thinking on capacity in the health sector. References to community capacity are found
mostly in literature on community
empowerment and strategies for improving
community mobilization and participation
(Goodman et al. 1998; Israel et al, 1994; Israel et al. 1998; and Eng and Parker, 1994).
The inputs in this framework represent the
resources available to individuals and communities. They include individual/family
factors, community factors, and factors outside the immediate influence of the community, such as exposure to health and education
programs. Processes explain how individuals
and communities use their resources to act in
support of their own capacity development.
Capacity outcomes relate to knowledge, motivation, skills and behavior that support individual and the communitys health and wellbeing. Performance is the actual behavior on
the part of individuals or communities that
might include interaction with the health system (participation or advocacy), as well as
behavior that directly influences health outcomes: utilization of health services, self
treatment, compliance, prevention behavior.

nel, and organizations cannot function without health personnel. Without individual users
of health services, the other levels cannot begin to perform effectively. Going beyond onedimensional diagrams to understand the dynamics of capacity building at each level and
between levels will guide the development of
M&E strategies and techniques.
For example, the processes listed at the system level in practice are often activities carried out by the MOH with support from donors and in collaboration with other actors in
the health sector (e.g., NGOs, private companies). There is a clear overlap between system
and organizational capacity since the capacity
of the system to carry out certain functions
may depend directly on the capacity of the
MOH to play its organizational role effectively. An M&E plan should attempt to
monitor changes at both levels to explain capacity development (or lack of it) well.

Using These Conceptual

Frameworks

The overview diagram that describes the relationship between capacity, performance and
sustainability also suggests a logical progression from capacity to performance to sustained performance, when in fact both capacity and performance can improve or decline in
uncoordinated or illogical ways. Because capacity is a fluid notion that responds to many
influences, linear frameworks, often used in
research and evaluation, are sometimes considered too mechanical for monitoring and
evaluating capacity. Cause and effect chains
related to capacity are seldom linear, suggesting the need to break out of a rigid, inflexible way of thinking.

While it is useful to separate levels of capacity for facilitating M&E planning, these levels
are clearly interdependent, as shown in the
nesting of health personnel and organization
levels in the system level, and the arrows
connecting individuals/communities to the
health system and its parts. A health system is
made up of organizations and health person-

Figures 2 5 suggest one way to look beyond

the linear representation of capacity variables
by depicting the process of capacity development as a cycle. Once one stage of capacity
development is achieved, capacity outcomes
become the new inputs and processes for the
next stage of improvement. Indicators in this
sense become relative, in that an indicator of

Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual Framework

25

capacity expressed as an outcome might be

described as another type of variable as capacity improves or declines.
This guide recommends the development of
conceptual frameworks as a useful process for
thinking through a capacity-building intervention strategy, clarifying expectations of
stakeholders and in hypothesizing the variables that are considered important to program results in a specific context. However,
these tools should be used along with strategies such as creative thinking, revisiting assumptions, and reflecting on results with
stakeholders when conducting capacitybuilding M&E. Part Three of the Guide will
elaborate on the use of frameworks or maps in
M&E and discuss these and other strategies
for understanding changes in capacity and
their relationship to performance.

Summary for Managers and

Evaluators

The first step in developing a vision of

capacity development, and a plan to

measure it, is to understand the role capacity plays in the health sector in developing countries.

We have found that directed discussion

using conceptual frameworks or maps

prior to M&E planning can stimulate
strategic thinking within project or work
teams and clarify individual and collective
expectations, and thereby improve capacity-building M&E.

26

The conceptual frameworks (Figures 1

5) illustrate the critical role capacity plays

in influencing and sustaining performance
in the health sector, including the four
levels where capacity is needed in the
health sector: system, organization, health
personnel and individual/community.

Figures 2 5 depict capacity at each level.

The purpose of these frameworks is to

show how capacity can be broken down
into inputs, processes, outputs, and outcomes in order to identify the different
factors that contribute to capacity and performance, and hypothesize about the potential relationships among these factors
within a single level.

The frameworks provide a starting point

for identifying the key variables that influence capacity and performance at that
level, and will help evaluators define capacity variables to track in the M&E plan.

Part 3

Monitoring and Evaluating Capacity-Building Interventions

Part 2 described a generic conceptual framework for understanding the role of capacity in
the health sector and suggested possible capacity variables for each level. This part presents the six steps for developing a monitoring
and evaluation plan for a specific capacitybuilding intervention. At the heart of this process is the development of a capacity map
or conceptual framework that applies to the
particular capacity-building intervention under study. The six steps are listed in Box 4.
Ideally an M&E plan should be formulated
during the design and planning of a capacitybuilding or performance improvement intervention. Evaluators and program planners
should work together with key stakeholders to
conduct a needs assessment, define the intervention strategy, and construct an M&E plan.
Since capacity building is often one strategy
in a broader approach to improving performance, capacity-building M&E should fit into
the overall performance-monitoring plan.

An M&E plan for capacity building states

what is to be evaluated, what evidence is
needed to answer key evaluation questions,
how the data will be used, who will use the
data, and for what purpose. The intended result of the planning steps is a clearly defined
guideline for data collection, analysis, and use
for assessing the effectiveness of a capacitybuilding intervention. In general, capacitybuilding M&E plans contain the following:

a conceptual framework
a definition of essential variables of capacity and performance
hypotheses on important links between
these capacity and performance variables
identification of the stages of capacity
indicators, and methods
a timeframe, and
a dissemination strategy

Box 4: Six Steps for Developing a Capacity-Building M&E Plan

1.

Define the purpose of the evaluation

2. Define performance objectives

3. Map capacity: Build a conceptual framework for the specific capacity-building
intervention
4. Identify capacity indicators
5. Identify appropriate methodological approach and sources of data
6. Develop an implementation and dissemination plan

Monitoring and Evaluating Capacity-Building Interventions

27

STEP 1

Define the Purpose of

the Evaluation

There are different types of evaluation, each

with a different purpose. In designing an
evaluation strategy, the evaluator first needs
to identify the key question(s) that he/she
wishes to answer and thus the type of
monitoring or evaluation to conduct. Table
3 illustrates some of the research questions
addressed by different types of capacitybuilding M&E.
A second question to address at the outset of
planning is: who are the intended users of
evaluation results? M&E of capacity-building
interventions can be used for different pur-

poses and to meet the needs of many different

stakeholders. It is advisable to specify the
primary and secondary users at the outset of
planning to avoid confusion and aggravation.
In the NGO Networks for Health Project, the
project partners and the donor expected to use
capacity-building monitoring data in different
ways. The NGOs sought information to
monitor the results of detailed internal organizational capacity-building plans. The funding
agency desired information on more general
capacity changes related to the quantity and
focus of programming in order to demonstrate
the overall results of the project. Until the
main purpose of collecting data was specified,
it was impossible to define the methods or
indicators in the M&E plan.

Table 3: Questions Posed by Different Types of Capacity-Building M&E

Type of Evaluation

Key Questions Answered

Needs assessment

What is the current level of capacity?

Where are the gaps in performance and capacity?
What capacity is needed?
How can the intervention best address the gaps in capacity and performance?

Monitoring

Inputs: Are inputs available to the program in appropriate quantities

and at appropriate frequency? Did the type or quantity of inputs
change?
Processes: Are key processes carried out to an acceptable standard or
at an acceptable frequency? Did the processes change?
Outputs: Are products related to capacity available? Did the products
expected emerge or change?
Outcomes: Is capacity appropriate and adequate? Did capacity improve?
Performance: Is performance appropriate and adequate? Did performance improve?

Evaluation

28

Did the capacity-building intervention lead to changes in capacity

and/or performance?

In practice, one finds an inherent tension in

defining the purpose of capacity-building
M&E. Managers generally use capacitybuilding M&E results for two main reasons.
The first is primarily an internal function, that
is, improving capacity and capacity-building
strategies. The second is primarily an external
function, that is, reporting on the progress of a
capacity-building intervention to various funders and other external stakeholders. While
the two purposes are not mutually exclusive,
managers must guide the M&E process care-

fully to ensure the best possible outcome. Too

much attention to serving external (often donor) needs has been found to dilute the use of
M&E for improving capacity-building strategies and organizational learning (Horton,
2001; Morgan, 1997). Lack of attention to
valid measures of change (or relying too
much on self-reported perceptions of capacity) can undermine the credibility of evaluation results. Box 5 summarizes key advice on
constructing a capacity-building M&E plan.

Box 5:DO'S AND DONTS of Developing an M&E Plan for a Capacity-Building

Intervention
DO
Develop capacity-building M&E plan during the intervention design phase

Develop capacity-building M&E plan with respect to broader performance objectives

Involve all stakeholders, both internal and external, in developing the M&E plan, particularly the purpose of the evaluation

Be prepared to negotiate with stakeholders on the purpose of the evaluation and make
all expectations transparent

DONT
Base M&E plans only on the needs of external stakeholders (mostly donors) at the expense of meeting internal information needs

Miss opportunities to reflect and learn about capacity development through M&E

Monitoring and Evaluating Capacity-Building Interventions

29

STEP 2

Define Performance
Objectives

Before launching into monitoring and evaluation of any capacity-building program or intervention it is critical to step back and fully
understand its focus and strategy. It is particularly crucial to understand how the stated
capacity-building strategy is expected to improve performance and what signs of improved effectiveness are expected from capacity building. Although it is not possible to
prove causality, it is important to clearly define the expected pathways between capacity
building and performance.
To begin, evaluators should address the following questions:

What is the purpose of the capacitybuilding intervention?

What type of performance is expected
in a given period and at what level:
health system, organization, health
personnel, or community?
What processes or activities are being
used to build capacity?
What external influences should be
taken into consideration?
Who has a stake in capacity building
and capacity measurement?

Defining Performance
Performance objectives should relate to the
mandate or specific purpose of a system, organization, or community, or to health personnel functions. The more specific one can
be about performance expectations, the easier
it will be to construct a capacity map. If the
M&E plan is being developed after a capacity-building intervention has been designed,
then articulating the performance focus and
expectations should not be difficult (assuming
the design document is sufficiently explicit
about performance objectives). Moreover,
some organizations already may adhere to a

30

set of performance indicators for internal

monitoring or reporting to external
stakeholders. Thus, there may already be
clearly stated performance standards. If, however, M&E planning takes place as part of the
design process (starting with needs assessment and intervention design) then focused
discussion among program planners, managers, and evaluators about what would constitute adequate performance in this context will
be needed.
In practice, perceptions of performance can
vary widely among stakeholders. For example, a manager of a clinic may define performance in terms of benefits to the clients;
whereas the clinics financial managers might
define performance as the acquisition of new
clients (and a correlating increase in income).
There is a growing body of literature about
Performance Improvement in the health sector, particularly organizational performance6
that can be useful for defining performance
expectations and identifying gaps in performance and possible reasons for those gaps. Performance objectives should be expressed as
variables or indicators that can be measured
against international or national standards, or
locally determined expectations. Normally,
the definition of performance objectives reflects both external and internal criteria. See
Box 6 for characteristics of a good performance objective and two examples of performance objectives that will be used to illustrate
capacity mapping in Step 3.

See Lusthaus, C., M. Adrien, G. Anderson, and F.

Carden. 1999. Enhancing Organizational Performance: A Toolbox for Self-Assessment, Ottawa: IDRC;
http://www.pihealthcare.org; McCaffrey, J., M. Luoma,
C. Newman et al. 2000. Performance Improvement:
Stages, Steps and Tools, Chapel Hill, NC: INTRAH.

Box 6: Characteristics of a Good Performance Objective

Measurable
Reflects a needed change
Relates to a clear product or action
Relates to a defined target population
Performed by specific delivery agent (e.g., organization, community group, etc.)
Relevant to a particular context/situation
Examples

Consistent delivery of a package of family planning services by X organization to a defined

population (defined in terms of coverage, quality, and consistency)

Improved demand for immunization services in communities served by community health

workers (CHW) (defined in terms of utilization and coverage)

Monitoring and Evaluating Capacity-Building Interventions

31

STEP 3

Mapping Capacity:
Build a Conceptual
Framework for a
Specific CapacityBuilding Intervention

Once performance objectives and expectations are defined, planners and evaluators
must make assumptions about the capacity
required to meet these objectives. Capacity
mapping is a structured process of thinking
through the role capacity plays in ensuring
performance by developing a conceptual
framework that is specific to a particular capacity-building intervention. During capacity
mapping, all the possible factors of capacity
that influence performance and the relationships between them must be identified. Once
the factors are all laid out, the program staff
or evaluator can focus on those that are most
essential for the evaluation.
Mapping capacity can be a critical step in
developing an M&E plan. The map is a tool
that guides the design of the plan, from selection of indicators and methods to presentation
of evaluation results. As stated by Morgan
(1997), evaluation designers and their program partners need a sense of what capacities they need to develop and for what reason.
Most groups and organizations can articulate
such a vision of the future given sufficient
time and productive discussion. Mapping
capacity makes plain to all stakeholders assumptions about key variables that affect the
desired outcome of a capacity-building intervention. A mapping exercise is an excellent
way to bring all stakeholders to a common
understanding of the scope and focus of a
capacity-building intervention, the performance outcomes expected from capacity development, and the role of M&E in tracking and
influencing change.
For the evaluator, the objective of this stage
of M&E planning is to create a conceptual

32

framework that links capacity-related inputs,

processes, outputs, and outcomes to performance of a system, organization, health personnel, or community. The advantage to the
evaluator of developing a capacity map is
twofold. First, through mapping, the evaluator
gains a better understanding of how key decision-makers and stakeholders believe the
system, organization, health personnel, or
community should be working. Second, mapping enables evaluators to define exactly
which capacity variables are to be evaluated
over time.
When to Map Capacity
As noted above, an M&E plan should be formulated during the design and planning of an
intervention. If program planning and M&E
design are conducted simultaneously, capacity
mapping can contribute to the choice of intervention strategies and to the M&E strategy.
However, sometimes circumstances do not
permit this ideal type of coordination on
program and M&E design. Frequently,
evaluation designers are brought in well after
program planners have defined the intervention strategy and specific activities. In this
case, evaluators must still work with program
planners to understand the intervention strategy and the role of evaluating it. Capacity
maps should reflect and/or inform this overall
strategy. If a conceptual framework already
Box 7:The Process of Capacity
Mapping
1.

Identify primary level of capacity

building
2. Define outcomes for that level
3. Develop a one-dimensional level
capacity map
4. Develop a multi-dimensional level
capacity map

exists for the intervention, designers should

review the assumptions and relationship
among variables depicted in this diagram to
understand the expected role of capacity
building. If an overall conceptual framework
for the intervention does not already exist, it
is essential to construct one to support capacity mapping.
How to Map Capacity
The process of developing a capacity map is
outlined in Box 7. During this process, planners, evaluators, and key stakeholders might
like to use the series of questions in Box 8 to
guide discussion. At a minimum, they should
consider the following two questions:
1. At which level is capacity required to ensure the stated performance objectives?
In other words, what level is likely to be the
main focus of capacity-building efforts? The
generic capacity map (Part 2, Figure 1) defines four different levels where capacity is
needed in the health sector: system, organization, health personnel, and individual/community level. Careful definition of the
performance objectives in Step 2, and a clear
understanding of the capacity-building strategy should help evaluators answer these
questions. For example, if performance gaps
are found in a specific health facility, then it
is likely that capacity-building interventions
will seek to improve capacity outcomes at the
organization or individual level. The first map
would focus on one of those levels.
2. What capacity outcomes are expected at
that level to improve performance?
Once the level has been specified, designers
should identify aspects of capacity that might
influence the specific performance objective
at that level and express them as capacity outcomes. Morgan (1997) defines capacity outcomes as the product of new learning and
abilities that eventually become part of the
organization or system, and support new lev-

els of performance. Designers can refer to

guides on organizational capacity development, for example, to help guide the choice of
capacity outcomes. However, capacity outcomes should always be tailored to performance objectives or standards of the particular
intervention or organization under study.
At the intervention design phase, it is worth
casting a wide net to consider all possible
aspects of capacity that might relate to desired
performance. Brainstorming on capacity can
then lead stakeholders or participants in this
mapping process to begin to prioritize areas
for capacity-building intervention. Where
parameters of an intervention are already set
or where a structure for brainstorming is
needed, designers might choose two or three
different areas of capacity development, express them as capacity outcomes, and then
map them. Although capacity building often
tries to address multiple capacity gaps simultaneously, for measurement purposes, it is
advisable to choose a limited number of key
capacity outcomes for capacity mapping.
For example, in Maps 1-3 below, the performance objective for the (fictitious) Family
Health Organization is defined as consistent
delivery of a package of essential, goodquality family planning services to a defined
population. Performance variables might
include coverage, quality, and consistency,
which would be expressed as indicators. The
three key capacity outcomes for this specific
performance objective are defined as financial
self-reliance, quality assurance practices institutionalized, and health services able to
respond to client needs. Although many other
aspects of capacity might influence coverage,
quality and consistency in the delivery of
family planning services, this organization
has chosen to concentrate on these three areas.

Monitoring and Evaluating Capacity-Building Interventions

33

Box 8: Questions to Guide Discussion for Capacity Mapping

Describing the link between capacity and performance
What elements of capacity are needed to ensure performance?
Where are the capacity gaps?
What might be the cause of poor capacity?
What are two or three key aspects of capacity required for performance?
At what level is capacity required?
Identifying capacity variables
What essential inputs and processes contribute to capacity at that level?
Describing the process of capacity development
Could capacity develop in stages?
How would one define possible stages of capacity?
What benchmarks might be used to mark these stages?
How would stages of capacity development manifest themselves in terms of improved
performance?

Single-Level Capacity Mapping

Once the two questions about levels and outcomes have been answered, it is necessary to
draw up a table or matrix that maps each capacity outcome at a single level. The process
involves identifying the variables that influence the specific capacity outcome at that
level. Capacity variables include inputs, such
as physical and human capital (defined by
Morgan, (1997) as knowledge, infrastructure
and skills) and processes representing
changes in human behavior (such as growth
of new skills, attitudes, values, and relationships) that are reflected in the functions performed by individuals or groups. These inputs
and processes come together to produce improved capacity outputs and outcomes. It is
often expected, in the course of capacity development, that individuals or groups add to
or build on their existing assets to make positive changes with respect to managing those
assets. A capacity map tries to capture these
critical assets and behaviors and link them to

34

capacity outcomes and new levels of performance.

Once completed, the map illustrates conceptually the pathway to achieving desired performance results. It includes specific variables
that may be targeted for intervention and then
monitored over the course of the intervention
to understand changes in inputs and processes
and any resulting improvements in capacity
outcomes. Evaluators are reminded that the
variables depicted in the capacity map are
those that relate to the inherent or desired
capacity of the system, organization, health
personnel, or individual/community targeted
for intervention. They do not represent elements of the capacity-building intervention
itself.

Box 9: Guidance on Capacity Mapping

Capacity mapping should refer to the logic of the overall program, project or intervention. Horton et al. describe this approach as referring to a theory of action & that binds interested parties into a single vision (Horton, 2001). Whether mapping capacity during intervention design or in
the context of an already defined intervention strategy, it is advisable to refer to existing data on
the intervention area, including needs assessment, capacity assessments, etc.

When mapping capacity it may be helpful to refer to the conceptual framework in Part 2 for a
general review of the role capacity plays in improving performance in the health sector and examples of capacity variables.

Be realistic about your expectations of the role of capacity. There is a tendency to consider
every aspect of resources and behavior in an individual, organization, or system as a capacity variable, and to risk measuring too much.

Look beyond individual capacity and training solutions to identify capacity variables. For example, during discussions on the capacity framework with SAIDIA, a Kenyan NGO (nongovernmental
organization) that provides health services and community development opportunities, staff at first
claimed that training health workers and community members was their only work in capacity building. Yet, with further discussion, participants illustrated a wide range of capacity-building activities
at all levels, including their work in coordination and collaboration with the public sector, and courting relations with donors that fund the NGO.

Map capacity with a wide range of stakeholders to inspire a sense of ownership of capacity
building and appreciation of the use of evaluation in programming. Since capacity-building M&E
delves into many internal characteristics and processes found within systems, organizations, and
communities, it requires considerable investment on the part of the members of these groups to
achieve success. The quality of information obtained from evaluation, therefore, is directly affected by the extent to which participants develop a feeling of ownership of the M&E activity and
value the data being collected.

To build such a capacity map, planners and

evaluators can use a facilitated discussion
among stakeholders as well as tap existing
data from needs assessments, capacity diagnoses and prior monitoring. Evaluators might
also draw on the experience of system and
organizational theory, theories of adult learning, and community development to hypothesize the most likely causes of poor performance. Box 9 provides some general guidance
for capacity mapping.

The following three diagrams (Maps 1, 2, and

3) provide examples of capacity maps that
define in a very general sense some possible
inputs, processes, and outputs related to the
three particular organizational capacity outcomes for the hypothetical Family Health
Organization: financial self-reliance, quality
assurance practices institutionalized, and
health services able to respond to client needs
and demands.

Monitoring and Evaluating Capacity-Building Interventions

35

Multi-Level Capacity Mapping

The three single-dimension capacity maps
provide a list of possible variables that influence capacity outcomes at one level. However, it is equally important to consider the
connections among levels where capacity
building might take place and their role with
respect to realizing capacity outcomes and
performance objectives. Although performance may be faltering at the facility, the strategy used to improve performance may require
additional capacity improvements at both the
health personnel and system levels. In this
case, designers may choose to construct a
capacity map that includes several levels and
that will provide even greater detail on possible variables that contribute to capacity outcomes. Thus, once the single-level map is
completed a second map is developed that
includes more than one dimension to illustrate
the interdependence among different levels of
capacity and determine which factors at other
levels might influence capacity outcomes at
the focus level. The two types of maps (single-level and multiple-level) will be used to
identify the variables to be assessed as part of
the M&E plan.
In Map 4, we have taken the same basic matrix but added a second axis to account for the
four possible levels of capacity. This example
focuses on the community level but the map
depicts variables at the four different levels
that might influence the specific communitylevel outcome.7 As noted in Map 4, the overall performance goal is to improve demand
for immunization services at the community
level, expressed as immunization service
utilization and coverage. The capacity of
Community Health Workers (CHW) to deliver IEC services was chosen as the capacity
outcome for mapping. In this case, the de7

This matrix is adapted from an exercise completed by

participants at a Workshop on Sustainability and Capacity Building hosted by PLAN International in May
2001 in Dakar, Senegal.

36

signers began by listing a large number of

possible capacity variables and then narrowed
them down to the key variables to be monitored over the course of the intervention.
Shaded areas represent an explicit decision
not to monitor an indicator in that category.
Dealing with Context
When assessing the effectiveness of capacitybuilding interventions it is also critical to understand the environmental or contextual
factors that influence capacity and performance. Horton and colleagues (2000) describe
context as formal and informal rules of the
game and how they are used. As noted in
Part 2, context can relate to the administrative, legal, political, socio-cultural, economic,
and technical forces that shape capacity and
performance. Clearly, many of these forces
are well beyond the reach of a typical capacity-building intervention. Nevertheless, it is
advisable for program managers to track environmental changes periodically. Organizational theory describes a successful (and sustainable) organization as one that understands
its environment and is able to adapt to environmental changes to ensure its survival.
Thus, tracking changes in the operational
context informs strategy for capacity development, even if planners or managers feel
there is little they can do to change it. The
publication, Enhancing Organizational Performance, published by the International Development Research Centre (IDRC) provides
a useful list of questions related to environmental influences on organizational capacity.
These questions are reproduced below in Box
10. In each map found in this guide there is
an additional box at the bottom where key
environmental variables are recorded.

Map 1: Organizational Capacity Map - Single Level

Capacity outcome: Financial self-sufficiency
Intervention
Performance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency).
Capacity-building objective: Improve financial self-reliance of health facilities in District One.
Strategies and activities: Improve leadership and financial planning skills of district managers; introduce new procedures for strategic planning; develop links
between health facilities and communities leading to joint planning and management; develop skills in grant application writing and reporting to funders.

Inputs

Processes

Outputs

Leadership

Strategic & operational

planning

Strategic & operational plans

developed and implemented

Financial management

Staff trained

Resource mobilization

Functioning financial
management system

Finances
Infrastructure
Human resources
Finance policy
Organizational culture

Human resource management

& development
Research, monitoring &
evaluation

Capacity
Outcome

Performance
Objective

Financial selfreliance (ability to

generate resources
& maintain a
healthy funding
base)

Consistent delivery of
essential package of goodquality family planning
services to a defined
population (coverage,
quality, and consistency)

External linkages established

(to donors, partners,
individuals, community)

Coordination with other

internal units
Creation & maintenance of
linkages with external groups
(specifically, funders)
Advocacy
Managing quality of care
Community mobilization

Context or operational environment

National policy on fee-for-service
National financial management procedures

Monitoring and Evaluating Capacity-Building Interventions

37

Map 2: Organizational Capacity Map - Single Level

Capacity outcome: Quality assurance practices institutionalized
Intervention
Performance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency).
Capacity-building objective: Improve quality assurance practices in health facilities in District One.
Strategies and activities: Improve leadership of facility managers and supervisors; introduce norms and procedures, clarify job descriptions and expectations;
improve links to supplies and logistics unit.

Capacity
Inputs

Capacity
Processes

Capacity
Outputs

Capacity
Outcome

Performance
Objective

Leadership

Operational planning

Financial resources

Human resource
management &
development

Operational plans
developed and
implemented

Quality assurance
practices
institutionalized

Consistent delivery of essential

package of good-quality family
planning services to a defined
population (coverage, quality,
and consistency)

Infrastructure
Human resources
Technology
Organizational culture

Staff, managers &

supervisors trained

Incentive practices
Research, monitoring &
evaluation

Quality assurance
standards clearly stated &
reference material
available

Logistics/supplies
management

Staff expectations clear to

them

Creation & maintenance

of linkages with other
organizations
(specifically, managers
and suppliers)

Monitoring reports on
quality, utilization, &
client satisfaction

Training and supervision

Functional relationships
between facilities and
suppliers

Context or operational environment

Published norms and standards for care
National health information system use of data to assess quality
Central stores policies and procedures

38

Map 3: Organizational Capacity Map - Single Level

Capacity outcome: Health services able to respond to client needs and demands
Intervention
Performance Objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality and consistency).
Capacity-building objective: Improve the ability of the health services to respond to client needs in District One.
Strategies and activities: Introduce incentives for quality of care practices; improve client provider communication skills; research and design optimal mechanisms for communication and interaction between communities and health facilities.

Capacity
Inputs

Capacity
Process

Capacity
Outputs

Capacity
Outcome

Performance
Objective

Leadership

Human resource
management &
orientation

Staff trained in technical

& communication skills

Health services able

to respond to client
needs and demands

Consistent delivery of essential

package of good-quality family
planning services to a defined
population (coverage, quality, and
consistency)

Finances
Infrastructure
Human resources
History of health service
organization
Organizational culture

Organizational incentive
practices
M&E, research
Coordination and
communication with
referral units
Creation & maintenance
of linkages with
community groups

Functional community
outreach &
communication
mechanisms
Feedback from routine
client satisfaction &
community monitoring
Quality of referral service
monitored

IEC
Community mobilization
Context or operational environment
National policy on consumer roles and rights
Published norms and standards of care

Monitoring and Evaluating Capacity-Building Interventions

39

Map 4: Community Capacity Map on Multiple Levels

Capacity outcome: Effective delivery of IEC services
Intervention
Performance objective: Increase demand for childhood immunization in Sierra Leone.
Capacity-building objective: Improve capacity of CHWs working with local NGO to provide IEC on childhood immunization.
Strategies and activities: Develop curricula for training of trainers and training of CHWs; conduct training of trainers and supervision; health personnel support
CHWs from health centers; NGO supervises and supports health center personnel working in service delivery.

Level

Capacity
Input

Capacity
Processes

Capacity
Outputs

Capacity
Outcomes

Training plan developed

Successful organization &

execution of training of trainers

System

National policy on
immunization and
community-based workers

Organizational
(Local NGO)

Health center personnel

(quantity/basic training)

Designing & planning

a training program

Community health worker

(quantity)

Supervision and
mentoring of CHWs

Curricula for:
Training of Trainers & for
Community Health
Workers

Participation in
Training of Trainers

Trainers meet standards

following course

Capacity of CHWs to deliver IEC

on immunization:

Participation in CHW
training on IEC

CHWs meet standards

following course

- CHWs skilled & motivated to

provide services

Performance

Training materials developed

Personnel

Ability to recognize training

needs and meeting them
Effective delivery of
IEC services (Quality
of IEC sessions)

IEC session provided

Community

Exposure to immunization
program

Community meetings
with CHWs

Level of participation in
health care learning activities
Recognition of need for
immunization

Context or operational environment

National economic growth
National health expenditures on immunization
Donor support for immunization

40

Community knowledge of
immunization benefits and side
effects
Caregivers value immunization

Improved demand for

immunization in
communities served
by CHWs
(coverage)

Interpreting and Using Capacity Maps

The examples of capacity maps above illustrate how the different factors of capacity
work together to drive or influence performance. They enable designers to view these
elements in a more systematic way that promotes common understanding and evaluation.
When capacity mapping is conducted after an
intervention has been planned, it can be used
to help evaluators understand the intentions of
managers in terms of their strategy for capacity development. During mapping, managers
are encouraged to pinpoint and define clearly
the areas of potential change that will serve as
indicators of progress in capacity development. Used after the design phase, the mapping exercise can reinforce existing capacitydevelopment strategies, thereby increasing
their specificity. Sometimes mapping can also
prompt planners to reexamine strategic
choices and change their tactics. Indeed, this
use of capacity mapping for strategic planning, and the linking of M&E with program
strategy should be encouraged throughout the
course
of
the
capacity
development/performance improvement intervention.

and performance. It should present capacity

variables in a general way. Planners and
evaluators then can discuss these variables
and narrow them down to priority areas of
intervention or measurement, and describe
them more specifically. The second or third
iteration of a map should be more precise in
depicting the variables to be monitored over
the course of the intervention. Map 5a provides an example of the first iteration of multiple-level capacity mapping. It contains a
wide range of general categories. Map 5b
illustrates the second iteration in which variables are specified in greater detail.
Through mapping, evaluators can identify and
organize the key questions to be addressed
regarding expected changes in the quantity,
quality, cost, and other key aspects of capacity which require monitoring over time. As
planners and evaluators interpret the map,
they will narrow down the focus of monitoring and evaluation activities. In Step 4, below, evaluators define indicators that measure
these variables and build them into a monitoring and evaluation plan.

Each type of mapping (single-level or multiple-level) can be done in two or three iterations. The first iteration of a map should attempt to provide a full list of capacity variables that may influence capacity outcomes

Monitoring and Evaluating Capacity-Building Interventions

41

Box 10:

Questions to Guide Discussion on the External Environment and Its

Influence on Organizational Capacity

Administrative
Is your organization influenced by the rule of other organizations, institutions, and groups to
which it is related or might be expected to be related?
Is your organization influenced by expectations of consumers, policymakers, suppliers, competitors, and other organizations in its external environment?
Are your organizations objectives and activities influenced by governments, donors, and
other organizations?
Is your organization influenced by important sector rules and regulations?
Do administrative norms/values in your country support or hinder the work your organization
intends to carry out?
Legal
Do the laws of the country support the role played by your organization?
Does the legal framework support the organizations autonomy?
Is the legal framework clear?
Is the legal framework consistent with current practice?
Is the legal regulatory context conducive to your organizations work?
Does your organization monitor changes in the legal context that could affect the position of
the organization?
Political environment issues
Do the political and ideological trends of the government support the kind of work the organization does?
Does the government system facilitate collaborative arrangements?
Does the organization play a role in national or sector development?
Does the organization have access to government funding?
Does the organization have access to international funding?
Does the organization have access to the governments knowledge and publications?
Do government policies and programs support the organization?
Sociocultural environment
Is equity in the workplace a social value?
Does the organization account for the effect of culture on program complexity?
Do values found in the sociocultural environment support the work of the organization?
Does the organization have access to a pool of capable human resources to recruit staff?
Does the organization analyze and link demographic trends to its work?
Economic environment
Does the governments economic policy support the organizations ability to acquire technologies and financial resources?
Is money available to do the organizations work?
Do donors support the organization?
Technological environment
Is adequate physical infrastructure (telecommunication, transport) in place to support the
organizations work?
Is the technology needed for your work supported by the overall level of national technology
development?

42

Does the government system facilitate the organizations process for acquiring needed technology?
Is the level of human resource development in your organization adequate to support new
technology?
Stakeholder environment
Is the community involved in the organization?
Are partners involved in the organization?
Do governments value the organizations products and services?
Do governments request or use the organizations products and services?
Do similar organizations compete or cooperate with your organization?
Do donors influence the organization?
Do funders support the organization?

The questions above are adapted from Enhancing Organizational Performance (Lusthaus et al.,
1999). While they are focused on the organization level, many of them can be adapted for any
level of the health system.

Monitoring and Evaluating Capacity-Building Interventions

43

STEP 4

Identify Capacity
Indicators

The next step in developing an M&E plan for

capacity building is to define indicators for
the elements of capacity identified during
capacity mapping. & Indicators are specific
variables that describe a given situation and
can be used to measure inputs, processes,
outputs, and outcomes at any level (system,
organization, health personnel, or individual/community). They can be constructed
from qualitative or quantitative data according
to the type of variable one is interested in
tracking. For example, the indicator number
of personnel per health facility trained in
control of sexually transmitted infections
(STI) tracks the inputs that influence capacity of a public health system. Alternatively,
measures of provider knowledge of appropriate treatment for different sexually transmitted infections and the availability of key STI
pharmaceuticals at each facility are outcome
indicators signaling capacity in service delivery. All three of these indicators could be
tracked to determine whether capacity exists
to meet system-level performance objectives,
such as quality of STI care.
What Are Capacity Indicators?
Capacity indicators generally project an aspiration or a sought-after state or ability. They
capture the current stock of resources available for various uses or an individual or organizational behavior that puts those resources into action (Moore et al., 2001). Defining or choosing indicators for M&E encourages planners and evaluators to be precise
about the inputs and processes that influence
capacity and performance and what types of
changes might result from capacity-building
interventions. Well-defined indicators provide
a reference framework for guiding all
stakeholders toward the same goals. Indicators also allow for standardized measurement
of change during implementation, which en-

44

ables evaluators to understand the process of

capacity development over time and its relationship to capacity-building intervention.
There is no agreed upon menu of standard
indicators of capacity development. As
Morgan (1997) states, It is difficult to find
useful examples of indicators that have been
used effectively to measure or assess capacity
building. Examples of common health sector-related indicators are found in the
MEASURE Evaluation Compendium of Indicators for Evaluating Reproductive Health
Programs (Bertrand and Escudero, 2002) and
other indicator handbooks. However, no single indicator manual focuses exclusively on
capacity building or differentiates between
capacity and performance measures. The obvious consequence is the need to work carefully and systematically during M&E planning to develop indicators that accurately reflect capacity development in each particular
context. Some capacity indicators can be
drawn from experience in human performance
improvement, organizational assessment and
theory, and other disciplines. Others will require testing through practice. When the
PRIME project developed an index of capacity in training organizations, it built on years
of experience working in this area and the
collective understanding of what it takes to
provide good-quality training on a sustainable
basis (Pyle and LaFond, 2001).
Even with the benefit of a generic indicator
reference material, most indicators used in
capacity-building M&E require some molding
or adaptation to a particular situation. For
example, if evaluators would like to study the
progressive stages of capacity development in
a specific organization, they might choose
indicators based on defined scales of organizational development, as in the Management
and Organizational Sustainability Tool
(MOST) developed by Management Sciences
for Health (MSH, 1996). However, they

should also adapt these indicators to a particular organizations baseline assessment of

capacity and its particular product or service.
Expectations for improved performance and
the timeframe of a specific capacity-building
intervention also matter. An organization pursuing capacity improvement in reproductive
health service delivery would choose different
measures of change from one seeking capacity improvement in networking and partnering. Thus, at the outset of M&E planning, one
should begin defining indicators based on the
capacity variables identified in mapping
rather than selecting indicators from a generic
list. Map 6 illustrates how indicators can be
added for each capacity variable, using the
format from Map 3. The discussion on indicators below begins with general guidance on
indicator design, provides examples of capacity indicators, and concludes with lessons
learned from a variety of capacity development experiences (in health and other sectors).
Working with Capacity Indicators
By now most program managers and evaluators at least have heard about what makes a
good indicator. In general, all indicators
should share the following traits:
Validity: Validity refers to whether the
indicator is measuring what it is supposed to
measure. Indicators should have a close connection with the intervention.
Reliability: Reliability refers to the degree of random measurement error in an indicator. Error may result from sampling or nonsampling; whether the response is inherently
objective or subjective.
Well-defined:
Indicator
definitions
should use clear and precise terms so everyone involved can understand what is being
measured.
Sensitivity: A sound indicator is sensitive
to the changes in program elements being
assessed.

Evaluators also need to take into account the

availability of data for operationalizing
indicators and the potential costs of gathering
data, in terms of financial resources and time.
Table 4 provides examples of health-sector
capacity indicators by level (system, organization, health personnel, and individual/community) and measurement variable
(input, process, output, and outcome) taken
from various sources (Morgan, 1997; Horton
et al, 2000; Bertrand and Escudero, 2002;
Brown, LaFond, and Macintyre, 2001). It
suggests wide variation in the indicators currently used to measure capacity and the need
for both quantitative and qualitative data
sources. The table is not intended to represent
relationships among these specific indicators.
Box 11 provides examples of capacity indicators used in non-health sector programs.
Table 5 gives examples of performance indicators at each level for reference.
Lessons for Indicator Development
The following lessons on indicator development are drawn from field experience in capacity measurement in health and other sectors (Morgan, 1997; Horton et al. 2000; Fort,
1999; Luoma, 2000; Ampomah, 2000; Catotti, 1999; Pyle and LaFond, 2001).

Lesson 1: Indicators should reflect an understanding of the change strategy for capacity development.
The process of choosing capacity indicators
should feed into the overall change strategy
designed for building capacity and improving
performance. Indicators should be developed
alongside capacity mapping while designing a
capacity-building intervention. Evaluators
also might seek to understand how information is currently used in the organization or
system to ensure that indicators become incentives for change and not barriers.

Monitoring and Evaluating Capacity-Building Interventions

45

Box 11: Examples of Capacity Indicators from Non-health Sector CapacityBuilding Interventions
Example 1
1. Capacity indicator related to decentralized payment functions administered by local officials,
district assembly members, and financial and political employees:
Ability of the system to transfer funds between authority levels (for example, within 45 days
of the end of the quarter) and/or produce audited statements within six months of the end of
the fiscal year.
2. Capacity indicator related to community water management committees role in water pump
maintenance:
A functioning Pump Management Committee that meets at least once a month and keeps the
pump functioning 90 percent of the time in normal circumstances.
3. Capacity indicator related to coordination of information among six ministries working on soil
erosion:
Twenty-five percent increase in the number of projects that require contributions from two or
more departments.
4. Capacity indicator related to government department to carry out joint surveys of client
farmers in delta area of cotton region:
Acceptance of survey methods as an effective tool by senior research officers and their incorporation into the work program of the agencies.
Source: Morgan, 1997
Example 2
Indicators related to motivation
Motivation to implement the strategic approach
Motivation to undertake strategic planning
Interest in improving the management information system
Interest in designing and managing competitive projects
Indicators related to capacity
Knowledge of the strategic approach
Skills to undertake strategic planning
Knowledge about designing and managing competitive projects
Knowledge about the foundations of an information management system
Indicators related to context or environment
Degree to which tasks demand conceptual and methodological creativity and innovation
Positive appreciation of performance in institutional evaluations
Degree of autonomy to undertake work
Contribution to improvement of the management information system
Source: Horton et al, 2000

46

Lesson 2: Capacity indicators should capture organizational and behavioral change

as well as material and technical change.

Lesson 4: Indicators should encourage

ownership and appreciation of the capacity-building and M&E process.

The most challenging demand of capacity

measurement is constructing meaningful
measures of human and organizational behavior change. There is a tendency, particularly in the health sector, to advance technical
explanations for what are just as likely to be
organizational or human behavioral problems.
For instance, it is often presumed that training
health providers alone will address performance gaps in service delivery when the root
causes of poor performance can range from
unreliable sources of supplies to low health
worker motivation. Capacity developers and
evaluators need to have a sense of how people
and organizations change, what brings about
lasting change, and why change in certain
values and practices makes a difference. Capacity indicators should capture the essence
of these changes in human and organizational
behavior.

Indicators should be designed to promote

ownership of the capacity-building process.
Evaluators should work with capacitybuilding stakeholders to define indicators that
reflect locally determined and accepted notions of change. Keeping indicator definitions
simple and relevant to local needs will encourage widespread use of M&E for capacity
development. Designing indicators to serve
external (often donor) needs rather than local
decision making can adversely influence
ownership of capacity development (Morgan
1997). This type of approach can diminish
the contribution that capacity indicators can
make to project effectiveness. Evaluators are
advised to balance the desire for more information for accountability purposes with the
value of using information to motivate positive behavior changes in individuals and organizations.

Lesson 3: In planning capacity-building

M&E, it is important to monitor not only
capacity but also key aspects of performance and the environment.

Evaluators should also keep in mind that

measuring capacity can also be a sensitive
issue. Organizations, and people, do not relish
having their weaknesses documented. They
feel even less enthusiastic about having their
weaknesses broadcast to their superiors, partners, and funders. The quality of data gathered for constructing capacity indicators could
be distorted and/or obstructed unless the purpose of monitoring and evaluation is clear to
all stakeholders, including the usefulness of
certain indicators. Indicators should be as
non-threatening as possible.

Improved performance serves as the main

reference for mapping capacity and is the goal
of capacity building. Evaluators should review changes in performance alongside capacity to examine the relationships among
different capacity and performance variables.
In addition, evaluators should track environmental changes. Environmental factors typically help to explain changes (or lack of
change) in capacity and performance. Indicators that monitor external conditions serve
as a warning to organizations that capacity
and performance may be in jeopardy.

Monitoring and Evaluating Capacity-Building Interventions

47

Map 5a: Mapping Capacity First Iteration

Intervention
Performance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency).
Capacity-building objective: Improve ability of health services to respond to client needs and demands in health facilities in District One.
Strategies and activities: Introduce incentives for quality of care practices; improve client provider communication skills; research and design optimal mechanisms for communication and interaction between communities and health facilities.

Inputs
System

Organization

Processes

Outputs

Outcomes

Performance

Supplies & delivery of essential

goods
Leadership

Supervisors

Quality of referral
system

Human resource

Incentives

Health services able

to respond to client
needs and demands

Supplies

Referral

Consistent delivery of
essential package of
good quality family
planning services to a
defined population (coverage, quality, and consistency)

Civil service administration

practices

Feedback
Supplies management
Personnel

Number of staff

Outreach
Learning

Community

Experience with family planning

Provider-client interaction
Links to community

Number of contacts

Local health organizations

Leadership

Context or operational environment

National policy on consumer roles and rights
Published norms and standards of care

48

Outcome of contacts

Map 5b: Mapping Capacity Second Iteration

Capacity outcome: Health services able to respond to client needs and demands
Intervention
Performance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency).
Capacity-building objective: Improve ability of health services to respond to client needs and demands in health facilities in District One.
Strategies and activities: Introduce incentives for quality of care practices; improve client provider communication skills; research and design optimal mechanisms for communication and interaction between communities and health facilities.

Inputs
System

Processes

Outputs

Outcomes

Performance

Leadership within management

teams with knowledge and training
in family planning

Behavior of supervisors (content,

communication & modeling of desired
behavior among health workers)

Number of commodity reports

Human resource (quantity & quality

of existing training/skills)

Incentives for supervisors & providers to

perform adequately

Client feedback on services

Health services able to

respond to client needs
and demand:
(Expressed as: Utilization; Client satisfaction;
and Supplies availability/
stockouts)

Consistent delivery of
essential package of
good-quality family
planning services to a
defined population (coverage, quality, and consistency)

Supplies of family planning and IEC

materials (quantity & reliability)

Referral system (designating, enabling &

following up referrals)

Supplies management checklist used

Civil service administration practices that support counseling and

provision of family planning
Supplies & delivery of essential
goods family planning supplies

Organization

Worker feedback on supervision

Frequency of needed referral

Personnel

Number of staff in each professional

category related to family planning

Community outreach activity

(frequency and quality)

Number of outreach visits

Availability & use of learning opportunities for improving communication on

family planning

Health workers motivated to address client

needs
Health workers ability
to conduct client interview

Provider-client interaction index (quality)

Community

Experience with family planning

Mechanisms for linking health services &

community groups (frequency & quality)

Number of contacts with

health facilities

Local organizations/unit focused on

health

Outcome of contacts in
terms of client satisfaction

Leadership

Context or operational environment

National policy on consumer roles and rights
Published norms and standards of care

Monitoring and Evaluating Capacity-Building Interventions

49

Map 6: Community Capacity Map on Multiple Levels with Indicators (in Italics)
Intervention
Performance objective: To increase demand for childhood immunization in Sierra Leone.
Capacity-building objective: Work with a local NGO to improve Community Health Workers (CHW) capacity to provide Information, Education, &
Communication (IEC) on childhood immunization.
Strategies and activities: Develop curricula for training of trainers and training of CHWs; conduct training of trainers and supervision; health personnel support CHWs from health centers; NGO supervises and supports health center personnel working in service delivery.
Level

Capacity Inputs

System

National policy on
immunization & CHWs
(Policy exists & is favorable)

Organization
(Local NGO)

Health center personnel

(Quantity/ basic training)
Community health workers
(Quantity/ basic training)

Capacity Processes

Capacity Outputs

Capacity Outcomes

Designing & planning a

training program
(Planning mechanisms
exist & planning skills
demonstrated)

Training plan developed

(Plan exists)
Training materials developed
(Quantity/quality of materials)

Successful organization &

execution of Training of
Trainers (TOT completed;
trainees knowledge improves;
trainees satisfied)

Performance

Ability to recognize training

needs and meet them
(assessment process leads to
training)
Personnel

Community

Curricula for:
Training of Trainers
and
Community Health
Workers
(curriculum exists)

Exposure to immunization
program (Past experience
with childhood
immunization)

Participation in Training
of Trainers

Trainers meet standards following course

(Post-test scores)

Capacity of CHWs to delivery

IEC on immunization

Participation in CHW
training on IEC
(% of personnel or
CHWs completing
training)

CHWs meet standards following course

(Post-test scores)

- CHWs motivated to provide

services (attitudes of CHWs to
IEC)

IEC sessions provided

(Number/frequency of IEC sessions)
Perceptions of CHWs
(Community relationship with CHWs and
acceptability of their role)

Community knowledge of
immunization benefits and side
effects (Index of immunization
program message recall)

Context or operational environment

National economic growth (GDP)
National health expenditures on immunization (% of health budget spent on immunization; total expenditure on immunization)
Donor support for immunization (% of immunization expenditure from external sources)

50

Effective delivery of IEC

services (Quality of IEC
sessions)

Improved demand for

immunization services in
communities serviced by
CHWs (Immunization
service utilization &
coverage)

Lesson 5: The results of indicator-based

capacity-building M&E should be interpreted wisely.

indicators based on program objectives and

develop a manageable set to monitor over
time.

There are documented challenges to using

indicators to monitor and evaluate capacity
building. Evaluators can manage each challenge with careful planning of M&E. Some of
these challenges are detailed below.

Evaluators are experimenting with indices or

complex indicators that combine a short list
of essential indicators (sometimes weighted
by strength of influence) into a single measure
of capacity. Of the few examples in the health
sector, the PRIME project used a single index
to assess capacity dimensions of organizations
that conduct training in reproductive health
(Fort, 1999; Ampomah, 2000; Catotti, 1999).
This index also takes into account different
possible stages of capacity by using a scale
from 0 to 4 to assess progress of an organization for each indicator under study. An example of the indicators and scales used in the
Training Organizations Index and a presentation of the results of a capacity assessment in
El Salvador are found in Annexes A and B.
The PRIME Project did not use this index to
conduct routine monitoring and evaluation of
training organizations; however, it has
adapted many of these indicators and the
scaling approach for use in its performance
monitoring plan (PRIME II, 2001). Other
examples that use scales or scoring as part of
a capacity index can be found in the Management and Organizational Sustainability
Tool (MOST) (MSH, 1996), and tools developed to evaluate the capacity of agricultural
research organizations (Horton et al., 2000).8

Capacity development is context specific. It

reflects qualitative (as well as quantitative)
changes in resource availability and behavior.
Given the wide range of possible scenarios
and capacity/performance objectives, it is
often not possible to establish objective standards that would allow local or regional comparisons in capacity across similar types of
entities. Internal benchmarks can be set, but
they may not be valid for other entities or
contexts. It follows that aggregation of indicators on a district, regional, or national scale
is not likely to result in useful information for
M&E.
Selection of capacity indicators is often
highly subjective. To encourage ownership
and relevance, evaluators often rely on perceptions of capacity and capacity change
among participants in the capacity development process as the basis for measuring progress. Thus, there is a need to balance these
subjective measures with a range of objective
indicators and data-gathering strategies.
Capacity is influenced by many different
variables. Hence, there is a tendency to try to
monitor a number of indicators at the same
time. We encourage the use of multiple indicators for each level within the capacity map
because they provide greater insights into the
state of capacity and can serve to validate
findings. Use of multiple indicators is often
recommended to explain what can be an imprecise situation or occurrence. At the same
time, however, evaluators should prioritize

It is important to note that indices can be difficult to

interpret if they are presented out of context or to an
audience that does not understand how the index is
constructed.

Monitoring and Evaluating Capacity-Building Interventions

51

Table 4: Examples of Capacity Indicators in Current Use in Health Programs

Level

Inputs

Process

Outputs

Outcomes

Health system

Doctors per population

Ratio of health care spending
on primary health care vs.
tertiary care
Percent of health budget funded
by external sources
Percent of national budget
allocated to health
Existence of clear mission
statement
Number of trained managers
per unit
Percent of district medical
officers with public health
degree/training
Clearly defined organizational
structure
Organizational culture that
values and rewards
performance

Donor coordination committee meets

every 6 months
Collaborative arrangements exist
between social sectors e.g.,
meetings between health &
agriculture or health & education
Percent of districts with decentralized
budgeting
Coordination with other organizations
evident through internal reporting
mechanisms
Number & quality of jointly
administered activities with partner
organizations
Job descriptions updated regularly to
reflect real work requirements &
responsibilities
Team planning (frequency and
quality)
Supervisors playing mentoring role

Number of multisectoral meetings

held
Number of collaborative projects
initiated in sectors outside health
Existence of national standards for
professional qualifications
Existence of sector-wide strategy

Adequacy of training
materials/supplies has been
assessed in one or more
institutions
Adequate training supplies
available in sufficient
quantities to support ongoing
RH/FP training in one or more
institutions
Up-to-date curricula
Percent of training budget from
external assistance
Average level of education
attained in the district
Mean income level
Proportion of adults whose
partner recently died in central
hospital
Community leadership (type
and quality)

Number of training sessions to

improve human resource management
addressing needs expressed by
providers
Managers trained in and using
performance evaluation
Percent of courses where training
methodology is appropriate for
transfer of skills/knowledge
Professional networking (frequency
and quality)

Presence of financial management

system that regularly provides
income/revenue data & cash flow
analysis
Number of commodity tracking
reports
Individual work plans are prepared
for all staff
Sufficient number of sites
functioning as clinical training sites
to meet clinic practice needs
Percent of MIS reports complete and
on time
Number of providers trained, by
type of training & cadre of provider
Number of staff trained in finance,
MIS, strategic planning, financial
planning
Number of managers trained, by
type of training
Number of monthly staff newsletters
produced

Number of health committees who

meet regularly and take action
Percent of dispensary budget
supported with community-based
funding
Level of community cohesiveness
Community experience negotiating
with district health office

Proportion of non-users who desire

to use contraception in the future
Level of participation in community
health committees
Number of health action plans

Organization

Health
Personnel

Individual/
Community

52

Widely distributed sector-wide

strategy
Regular auditing of system-wide
accounts by independent company
Percent of recurrent costs covered
through local resource generation

Supervisors able to guide on-site

learning
Ability to adjust services in
response to evaluation results or
emergencies
Cost-sharing revenue as a
proportion of the annual MOH
non-wage recurrent budget
Percent of facilities with stock-out
of essential commodities in the last
6 months
Regular review of MIS data for
routine planning
Percent of trainees (providers) with
knowledge in skill area (meet
national standard)
Level of staff motivation
Percentage of senior staff with
continuing education opportunities

Community needs presented to

district health office on regular
basis
Proportion who knows anemia
prevention practices
Level of community mobilization
and empowerment
Community support for
maintaining new well

Table 5: Examples of Performance Indicators in Current Use in Health Programs

Level
Health system

Organization

Health Personnel

Individual/
Community

Average time/distance to the nearest reproductive health facility offering a specific service
Percent of facilities where percent of clients receive the service that meets the expected standards
Number/percent of trainees deployed to an appropriate service delivery point and job assignment
Percent of facilities that experience a stockout at any point during a given time period
Percent of health facilities providing STI services with adequate drug supply
Contraceptive prevalence rate (CPR)
Disability adjusted life years (DALY)
Disability adjusted life expectancy (DALE)
System responsiveness to clients
Index of equality of child survival
Total health expenditure as a percent of GDP
Public expenditure on health as a percent of total public expenditure
Out of pocket expenditure as a percent of total health expenditure
Percent of mothers examined every 30 minutes during the first two hours after delivery
Percent of data elements reported accurately in MIS reports
Family planning continuation rates in catchment population
Percent of annual revenue generated from diverse sources
Percent of target population that received DPT 3 immunization
Cost of one months supply of contraceptives as a percent of monthly wages
Percent of deliveries in which a partograph is correctly used
Percent of newborns receiving immediate care according to MOH guidelines
Percent of pregnant women counseled and tested for HIV
Percent of STI patients appropriately diagnosed and treated
Percent of communities with active health center management committee
Percent of target population that received DPT 3 immunization
Percent of non-users who intend to adopt a certain practice in the future
Percent of infants 0 - < 6 months of age who are exclusively breastfed
Percent using condoms at last higher-risk sex

Monitoring and Evaluating Capacity-Building Interventions

53

Determining cause and effect are not easily

done with capacity-building M&E, even
though a capacity map might clearly state
assumptions about relationships among variables. The multiplicity of capacity variables
and the frequent improvement and decline in
capacity make it difficult to draw definite
conclusions from a complex situation. It is not
surprising, therefore, that some evaluators
have found linear evaluation frameworks and
the strict use of indicators too inflexible and
mechanical to be used effectively in monitoring and evaluating capacity (Morgan,
1997; Earl et al., 2001). For these and other
reasons, Morgan cautions evaluators not to
rely too heavily on indicators to provide complete insights into capacity development. In
spite of the growing list of capacity measures,
indicators used in monitoring and evaluation
of capacity do not explain why complex systems work the way they do (Morgan, 1997).

54

In light of these challenges, the way in which

indicators are developed, measured and used
becomes a critical determinant of the credibility and usefulness of monitoring and
evaluation of capacity building. Many of
these constraints can be addressed with careful indicator development and the use of a
range of data-collection instruments that are
sensitive to the intangible nature of what is
being measured in capacity-building evaluation. At the same time, the use of linear
evaluation frameworks also requires careful
management. Evaluators need to focus on
critical process aspects of capacity building,
and use maps to guide but not restrict M&E.

STEP 5

Identify Appropriate
Methodological
Approach and Sources
of Data

The fifth step in developing a capacitybuilding M&E plan involves defining the
methodological approach, identifying sources
of data, and choosing (or developing) data
collection tools. Evaluators should ask the
following questions:
Which methodological approach is appropriate?
What sources of data are necessary for
measuring the indicators defined in Step
4?
Are there any existing tools for measuring
capacity that are appropriate for my purposes?
Methodological Approaches and Challenges
As discussed throughout this guide, monitoring and evaluation require different methodological approaches and have different data
needs. The choice of methods and data
sources relates mainly to the purpose of the
evaluation (see Step 1).
Is the purpose to monitor the implementation of a capacity-building intervention,
assess its effectiveness, or both?
Will the results be used mainly for internal improvements or external reporting?
Clearly, all capacity-building programs need
to be monitored to ensure they are working
well (i.e. to track changes in inputs, processes,
outputs and outcomes). However, the evaluation of program effectiveness happens less
frequently and only for selected interventions
due to cost and complexity. In the case of
capacity-building evaluation, it can be particularly difficult to conduct evaluations that
look for an association between capacitybuilding intervention and changes in capacity
or performance. These changes can occur for
a number of reasons in addition to the capac-

ity-building intervention itself (e.g. contextual

influence). Since capacity measures are not
easily quantified, and identifying similar organizations or systems to facilitate comparison (as in a case-control study) is difficult,
experimental designs are not feasible or practical for capacity measurement. As James
(2001) notes about capacity-building evaluation, precise measurement and attribution of
cause and effect is rarely possible and never
cost effective. The best we can hope for is
plausible association.
Evaluators are therefore advised to recognize
the challenges to capacity-building M&E and
set realistic aims for evaluation. Many of
these challenges have been discussed previously in this guide. Some of them relate to the
inherent nature of capacity (capacity and capacity building are dynamic and multidimensional; contextual), while others are a function
of the early stage of development of capacity
measurement. Four of the main challenges are
detailed below.
Capacity develops in stages
Capacity measurement tools should be able to
capture different stages of development of
communities, health personnel, organizations,
or health systems. The MSH organizational
profile used in the Management and Organizational Sustainability Tool (MOST), for
example, has identified different benchmarks
according to an organizations stage of development (nascent, emerging, mature). Capacity measurement must be able to capture individual elements of capacity and combinations
of elements, and relate them to the stage of
development of the entity being assessed.
Changes in capacity need to be measured
over time
Repeated measures are needed to capture the
interim steps in capacity-building processes as
well as trends in outcomes. While there are
examples of repeated application of capacity

Monitoring and Evaluating Capacity-Building Interventions

55

measurement tools (INTRAH, SFPS, and

PASCA), to date, only limited reports of
findings from longitudinal evaluations are
available (PASCA). Better techniques are
needed to capture the effects of capacity
building over time and elaborate the link between capacity development and performance
improvement.
Internal versus external validity
Capacity building should be a self-motivated
and self-led process of change. Evaluation
strategies that use self-assessment techniques
and locally determined benchmarks of progress inspire ownership of capacity development and increase the likelihood that evaluation results will be used. Nevertheless, there
can be a cost to this approach in terms of the
perceived validity of findings. External
stakeholders often prefer to measure progress
against performance standards (of either national or local origin) using standardized indicators to allow comparisons or a reference to
other similar types of capacity-building programs. Self-reported measures of capacity
may not meet the reporting expectations of
external stakeholders even if they support
better capacity development strategies. Box
12 describes the experience of one project in
using the two different approaches.

Lag time between changes in capacity and

changes in performance
It is very common to experience considerable
lag time between a capacity-building intervention and changes in capacity, as well as
between changes in capacity and changes in
performance. Timing of capacity or performance measurement should take into consideration these delays and consider interim
measures of change or longer timeframes for
M&E.
Tackling Methodological Challenges
Many of the tools and methods reviewed for
this guide were able to tackle challenges to
capacity-building measurement. Others provided useful lessons on how to move capacity-building M&E forward. Advice to evaluators follows:
use multiple data-collection instruments,
reflecting the multidimensional nature of
capacity. Multiple data-collection instruments are useful to get a comprehensive
picture of capacity or to assess capacity
from different perspectives (e.g., assessing
the views of managers and health workers
or assessing internal perspectives and
those of external examiners).

Box 12: PASCA: From Self-Assessment to External Assessment

PASCA is a USAID-funded project focusing on capacity building of nongovernmental organizations (NGOs) that provide HIV/AIDS services in Central America. During the first year of the
project (1996), PASCA conducted a self-administered needs assessment study among the NGOs
receiving support. Although the needs assessment provided useful information for planning, the
researchers felt that the self-administered methodology exaggerated the programmatic, administrative and managerial capacity of the NGOs. Thus, managers decided to conduct an externally administered Validation Study in 1997 using mixed methods to determine the validity of
the self-reported data, and provide an in-depth assessment of the management and programmatic needs of each NGO. When compared to the Needs Assessment survey, capacity scores
from the Validation Study were markedly different. The Validation Study, in which selfreported answers were validated with document observation, provided data that more accurately reflected the capacity of the NGOs (MEASURE Evaluation, 1998).

56

combine qualitative and quantitative

methods, such as focus groups, individual
interviews (with both closed- and openended questions), surveys, and document
reviews.

address more than one level. Capacity

often occurs at several levels simultaneously. New measurement tools are needed
to capture capacity building at a single
level and address the relationship between
levels.

include self-assessment techniques in

combination with external or standardized
methods. (See Box 13 for a discussion of
self-assessment and external assessment.)
Evaluators are urged to strike a balance
between meeting the need for evaluation
data that different stakeholders will deem
objective or credible, and promoting
performance improvement through monitoring and evaluation.

triangulate methods and data sources.

Triangulation examines results from a variety of data-collection instruments and
sources, strengthening the findings of capacity-building monitoring and evaluation. If all data lead to the same conclusion, then there is some confidence the result actually will reflect changes. Where
there is discordance in the results, it is
necessary to examine possible sources of
the differences. Looking at other sources
of data on similar topics can help understand findings as well.

use data interpretation workshops to obtain input from a range of stakeholders involved in the program (both internal and
external).

Sources of Data
A number of data sources are available for
monitoring and evaluating capacity building.
Since capacity measurement often includes
the use of multiple indicators, monitoring and
evaluation usually requires multiple data
sources. Indicator design should take into
account the potential availability of data particularly from existing sources. Organizations
and systems often have records and reports
that provide insights into different aspects of
capacity. Some examples of existing data
sources are presented below.
In many cases, however, it will be necessary
to collect new data to operationalize the indicators selected. As noted above, issues such
as data sensitivity (with respect to its effect on
validity), the purpose of monitoring and
evaluation, and the cost in terms of time and
resources required should guide evaluators in
determining what data will be collected and
how they will be collected.

Sources of data by level of capacity include:

System: national health policy records, national data-collection efforts (census, vital
statistics, national /regional surveys), international surveys (e.g., FPPE, API, DHS).9 MOH
policies, financial reports, legal or regulatory
statements (bills, acts, recommendations,
white papers, etc.).
Organization: routine health service records
and reports, budget and expenditure records,
financial statements, personnel records, program and donor reports, constitutional documentation, strategic and annual plans, meeting
minutes, evaluations and audits, organizational networking analysis, organizational
assessments.
9

FPPE (Family Planning Program Effort Score); API

(AIDS Program Effort Index); DHS (Demographic and
Health Survey).

Monitoring and Evaluating Capacity-Building Interventions

57

Health personnel: personnel records (job descriptions, performance evaluations, background checks, training summaries), supervision reports, self-evaluations.
Individual/Community: community-based and
social marketing surveys, community health
worker reports, meeting minutes, maps, focus
groups, and participatory appraisals.
In planning for data collection, it is often
helpful to develop a data chart that spells out
the key questions to be addressed, the indicator that links to the question, and the data
sources needed to answer the question. An
example of a data chart is found in Table 6.
Tools for Measuring Capacity at Different
Levels
A number of data-collection instruments and
tools have been developed and used to measure capacity at the four levels. (See Table 7
for a list of tools and their key characteristics). In most cases, these tools have been
used for capacity assessment rather than for
monitoring and evaluation. In addition, most
of the tools identified are designed to assess
organizational capacities, although many of

58

them assess the capacity of health program

personnel because of their central role in organizational functions and performance. We
identified a more limited number of tools to
measure the health system and individual/community level capacity. However, the
field of capacity measurement is changing
quickly and several agencies are currently
developing approaches to understanding
changes in performance at the system level
(Partnerships for Health Reform, 1997;
Murray and Frenk, 1999).
The tools listed in Table 7 are provided for
reference only. To determine if a tool might
be useful for a particular capacity development intervention, evaluators should address
the following questions:

At what level(s) do I want to assess capacity?

Do any of the existing instruments measure the dimensions or indicators I have
identified through mapping?
How could I adapt one of these instruments for my needs?

Box 13: Advantages and Disadvantages of Self-Assessment and

External Assessment Techniques
While practitioners value the role of self-assessment tools in stimulating interest in capacity
building and launching a change process, for monitoring and evaluation purposes it is important
to consider the potential advantages and disadvantages of both internal and external approaches.

Advantages of self-assessment tools:

Greater involvement of those whose capacities are being assessed (e.g., staff of an organization), which can lead to greater ownership of the results and, ultimately, greater
likelihood that capacity improvements (based on results of the assessment) will take
place
Non-threatening way to raise awareness of the importance of capacity improvement
among those involved in the assessment process
But self-assessment tools
Require an external facilitator
Rely on perceptions and may be less reliable when used repeatedly and are prone to various biases (e.g., optimistic bias)
Become less useful with high staff turn-over (which results in changing the self in selfassessment)
In many cases are interventions in and of themselves
Advantages of external assessments tools:
Often considered more objective

But external assessment tools

May be more costly due to the cost of external consultants; self-assessments, particularly those that require intensive facilitation, can also be demanding in terms of time
May not reflect internal views accurately
Recommendation:
Use a mixture of methods that combine subjective and objective measurement.

Monitoring and Evaluating Capacity-Building Interventions

59

Table 6: Example of a Table of Data Sources for an Organizational Assessment

M&E Question(s)

Objective(s)

Indicator

Method(s)

Data Sources

1.

Did financial and human

resource inputs change
over time?

1.

1.

Amount of budgetary
resources by source over
time

1.

Records review (organization

and donor)

1.

Accounts, budgets, annual

reports

2.

Did the source of

financial resources
change over time?

2.

Number of management
and staff positions filled
over time

Record review of personnel

resources

2.

2.

Personnel records, annual

reports

3.

Interviews with senior

management in organization
and donors/NGOs

3.

Finance manager,
accountant, donor/NGO
representative

2.

1.

Did the organization

establish new
relationships or improve
links with other
organizations that
contributed to achieving
performance objectives?

1.

Determine whether
capacity-building
interventions increased
budgetary resources of the
organization and the
number of trained
personnel.
Determine whether
change in reliance on
donor/NGO funding has
decreased.
Determine the extent of
networking and its effect
on organizational
behavior.

1.

Number of joint activities

with other organizations

1.

Prospective recording of links

to other organizations

1.

Record forms

2.

Frequency of contact with

higher and lower level
organizations within public
sector

2.

Interviews with management

and staff

2.

Questionnaire and focus

groups

3.

Facility survey (observation,

exit interviews, provider
interview, inventory)

3.

Survey data

4.

Organizational
networking analysis

1.

Facility survey (exit

interviews, provider interview)

1.

Survey data

2.

Focus group data

2.

Client focus groups

3.
1.

Provider focus groups

Facility survey (exit
interviews, provider interview)

1.

Survey data

2.

Focus group data

2.

Client focus groups

3.

Provider focus groups

3.

1.

1.

60

Did staff capacity to

assess client needs
improve?

Did staff capacity to

meet client needs
improve?

1.

1.

Determine the
effectiveness of training
and mentoring.

Determine the
effectiveness of training
and mentoring.

1.

Types and frequency of

outcomes from links with
other organizations
analyzed by organization
type (public or private)
Client satisfaction index

2.

Provider satisfaction index

1.

Client satisfaction index

2.

Provider satisfaction index

Table 7: Capacity Measurement Tools

Level

Methods

Self/
External
Assessment

Single/
Multiple
tools

IDRC

Organization

Qualitative and
quantitative

External and
self-assessment

Multiple

Outcome Mapping: A Method

for Reporting on Results
http://www.idrc.ca/telecentre/
evaluation/html/29_Out.html

IDRC

System
Organization

Qualitative and
quantitative

Self-assessment

Multiple

Integrated Health Facility

Assessment (IHFA)
http://www.basics.org/publica
tions/pubs/hfa/hfa_toc.htm

BASICS

Organization

Quantitative
assessment

External assessment

Multiple

Management and Organizational Sustainability Tool

(MOST)
http://erc.msh.org/mainpage.c
fm?file=95.40.htm&module=t
oolkit&language=English

Family Planning
Management
Development
(FPMD)/
MSH

Organization

Qualitative

Self-assessment

Single

Management Development
Assessment (MDA)
http://erc.msh.org/mainpage.c
fm?file=95.50.htm&module=t
oolkit&language=English

FPMD/MSH

Organization

Quantitative

Self-assessment

Single

The Child Survival

Sustainability Assessment
(CSSA)
http://www.childsurvival.com

Child Survival
Technical
Support (CSTS)
Project/ORC
MACRO
CSTS
Project/ORC
MACRO

System
(local)
Organization
Community

Qualitative and
quantitative

Self and internal

client assessment

Multiple

System
(local)
Organization

Qualitative and
quantitative

Self and internal

client
assessment

Multiple

Tool

Developed By

Enhancing Organizational
Performance: A Toolbox for
Self Assessment
http://www.idrc.ca

The Institutional Strengths

Assessment (ISA) Tool
http://www.childsurvival.com/
tools/project_planning.cfm

Monitoring and Evaluating Capacity-Building Interventions

Short description
Measures the results of an organizations programs, products and services and then integrates these results with the
techniques of formative assessment in which the assessment
team becomes involved in helping the organization meet its
goals.
Outcome Mapping characterizes and assesses the contributions a project or organization makes to significant and
lasting changes (outcomes). In Outcome Mapping a program is assessed against its activities that contribute to a
desired outcome, not against the outcome itself.
This manual outlines the key steps for planning and conducting an integrated health facility assessment at outpatient
health facilities in developing countries. This assessment is
designed for use by primary health care programs that are
planning to integrate child health care services.
The Management and Organizational Sustainability Tool
(MOST) is a package (instrument and user's guide) designed
to facilitate management self-assessment and to support
management improvement. MOST uses an instrument to
help focus an organization on the actual characteristics of
their management, identify directions and strategies for
improvement, and set priorities for the management development effort.
This tool includes four steps: 1) develop a preliminary
management map to guide assessment; 2) develop and
administer MDA questionnaire to collect information on the
management capabilities of organization; 3) analyze survey
results and develop a post-survey management map; and 4)
develop action plan for making improvements.
Evaluation framework to systematically measure progress
toward sustainable health goals. Process that projects can
use to lead a participatory assessment with communities and
local partners.
This self-assessment tool is currently being pilot tested by
CSTS.

61

Self/
External
Assessment

Single/
Multiple
tools

Qualitative and
quantitative

Self and
internal client
assessment

Multiple

Organization

Qualitative and
quantitative

Self-assessment

Multiple

World Vision

Community

Qualitative and
quantitative

External and
self-assessment

Multiple

Provides technical guidance for measuring the

Transformational Development Indicators. It includes 8
volumes that cover indicator definitions and methods for
collecting, analyzing, and reporting on the indicators.

Center for
Communications
Programs
(CCP)/Johns
Hopkins
University

Community

Qualitative and
quantitative

External and
self-assessment

Multiple

Presents model, process and outcome indicators, and some

data collection and analytical tools for use by communities.

Tool

Developed By

INTRAH/PRIME
Capacity Building In Training
Questionnaire
http://www.prime2.org/prime
2/techreport/home/50.html

Client-Oriented Provider
Efficient (COPE)
http://www.engenderhealth.or
g/ia/sfq/qcope.html
Note: COPE has now been
adapted for use with maternal
health services and community partnership
http://www.engenderhealth.or
g/news/newsreleases/020516.
html
Transformational Development Indicators Field Guide
http://www.worldvision.org
NOTE: Tool not yet available
online
Communication for Social
Change: An Integrated Model
for Measuring the Process and
Its Outcomes
http://164.109.175.24/Docum
ents/540/socialchange.pdf

62

Level

Methods

INTRAH/
PRIMEII

Organization

Engender Health

Short description

The framework and tool developed at the end of PRIME I

has been used to aid program evaluation in different
countries (e.g., Mexico, Ghana, India, and Bangladesh),
when interventions have focused on the strengthening of
training and service delivery institutions. The tool
encourages organizations to discover root causes of
obstacles with a sustainable effort to build capacity in the
organization to recognize, address, analyze and prioritize
problems.
COPE encourages and enables service providers and other
staff at a facility to assess the services they provide jointly
with their supervisors. Using various tools, they identify
problems, find the root causes, and develop effective
solutions.

Self/
External
Assessment

Single/
Multiple
tools

Quantitative

External and
self-assessment

Multiple
instruments

Scores organizational competence, commitment, clout,

coverage and continuity.

Organization

Quantitative

External and
self-assessment

Single
instrument

Tools are in Spanish only.

PASCA

Organization

Quantitative

External and
self-assessment

Single
instrument

Tools are in Spanish only.

Institutional Assessment
Instrument (IAI)
http://www.worldlearning.org
or
http://www.worldlearning.org/
pidt/docs/wl_instcape.pdf

World Learning
Project Inc.

Organization

Qualitative and
quantitative

External
assessment

Multiple
instruments

Provides a framework for assessing the institutional needs

of a single organization or a community of organizations.
Pinpoints six key areas generally agreed to be the components of effective institutions.

Institutional Development
Assessment (IDA)
http://www.fhasfps.org/documentsdownload/
Institutional%20Development%20A
ssessments.PDF
Organizational Capacity
Assessment Tool (OCAT)
http://www.pactworld.org

SFPS

Organization

Qualitative and
quantitative

External
assessment

Multiple
instruments

Documents existing capacity and identifies potential areas

of collaboration and capacity building in overall dimensions
of management, financial management and technical capacity.

Pact/Ethiopia

Organization

Quantitative

Self-assessment

Multiple
instruments

A methodology for organizational capacity assessment and

strengthening that helps organizations anticipate and overcome the greatest barriers to organizational change and
growth. Through a guided self-assessment and planning
process, organizations reflect upon their performance and
select the tools and strategies they need to build capacity
and broaden impact. A four-staged process that includes:
Participatory tool design; guided self-assessment; dataguided action planning; reassessment for continual learning
that allows organizations to monitor change, track the effectiveness of their capacity-building efforts, and integrate new
learning as their needs change and capabilities increase.

Tool

Developed By

Level

Methods

Assessing Institutional Capacity in Health Communication: A 5Cs Approach

Work in Progress.
http://www.jhuccp.org

CCP/Johns
Hopkins
University

Organization

Management/Financial
Sustainability Scale (MFSS)
http://www.pasca.org

PASCA

Systematic Approach Scale

(SAS)
http://www.pasca.org

Monitoring and Evaluating Capacity-Building Interventions

Short description

63

Tool

Developed By

Level

Methods

Participatory, ResultsOriented, Self-Evaluation

(PROSE)

Education
Development
Center and PACT

Organization

Qualitative and
quantitative

Self/
External
Assessment

Single/
Multiple
tools

Self-assessment

Single
instrument

SEE POET at:

http://www.undp.org/csopp/po
et.htm

National program effort

indices
Family Planning Effort Index
(FPEI)
http://www.agiusa.org/pubs/journals/271190
1.pdf
The AIDS Program Effort
Index (API)
http://www.policyproject.com
/pubs/countryreports/api.pdf

64

The Futures
Group/
Population
Council

System
(national)
Organization

Quantitative and
qualitative

External
assessment

Single
instrument

Short description
Participatory Organizational Evaluation Tool (POET) is an
organizational capacity assessment tool used to measure and
profile organizational capacities and consensus levels in
seven critical areas and assess, over time, the impact of
these activities on organizational capacity (benchmarking).
POET is based on a methodology called PROSE.
PROSE stands for Participatory, Results-Oriented, SelfEvaluation, a new methodology for assessing and enhancing
organizational capacities. PROSE is designed for use by
service organizations, schools, and government units. It is
suitable for assessing capacity and catalyzing organizational
change in relation to such concerns as: practices related to
exceeding customer expectations, organizational effectiveness in achieving mission, community participation, equity,
decentralization, and managerial effectiveness.
Each index measures national level effort and identifies
strengths and weaknesses of those efforts.

STEP 6

Develop an
Implementation and
Dissemination Plan

The final step in planning for capacitybuilding M&E is to develop an implementation plan to monitor and evaluate capacity. At
a minimum, the implementation plan should
include a timetable for data gathering and
review of data, individual responsibilities, a
dissemination strategy, and a budget. In practice, capacity measurement, as a reflection of
capacity development, is likely to be an iterative process rather than a perfunctory before
and after look at capacity. Experienced
evaluators (Horton et al, 2000; Lusthaus,
1999; Earl et al., 2001; Morgan, 1997) recommend regular review and discussion of
monitoring results with stakeholders to guide
the process of capacity development and encourage ownership of the monitoring process.
Setting aside enough time to present the results periodically and allow for discussion and
feedback from the stakeholders will greatly
enhance data interpretation and the impact of
the evaluation itself. As Morgan (1997) notes,
Indicators by themselves provide few answers. The information they produce must be

screened through the mental models of the

participants to acquire any diagnostic value.
When developed before the evaluation begins,
a dissemination strategy guides data collection and analysis. Developing a format for
presentation of the results to the appropriate
audience identifies weaknesses and gaps in
the evaluation plan. It also helps to guide the
direction of the evaluation by emphasizing
what is needed for addressing the needs of the
data users and raising awareness of possible
sensitivities. Gaps or excess data collection
becomes obvious, and further refinement of
the number or type of indicators being measured is often necessary. In the process, evaluators identify all key stakeholders that should
be alerted to the results, if they are not directly involved in the evaluation itself. The
recommended forum for disseminating results
is one that promotes discussion and interaction among the key stakeholders and those in
a position to influence the future direction of
the capacity-building efforts. Sufficient funds
must be set aside so that all those who make a
credible contribution to the evaluation receive
at least summary results in a timely and relevant fashion.

Monitoring and Evaluating Capacity-Building Interventions

65

Part 4

Summary Checklist: Steps for Designing a Capacity-Building

M&E Plan

This guide is designed to assist manager and

evaluators working in international healthsector capacity development to
gain a clear understanding of the concepts of capacity and capacity building
critically evaluate the strengths and
limitations of current approaches to capacity measurement
design a capacity-building M&E plan
that outlines a systematic approach to
measuring capacity and assessing the results of capacity-building interventions
in the health sector
The manual presents a discussion of the
concept of capacity and capacity building,
and the influence of attributes of capacity on
M&E approaches. It outlines a conceptual
framework for understanding the role that
capacity plays in enabling performance in
the health sector and suggests an approach
to identifying key factors that influence capacity and performance. Finally, it outlines
some basic steps for capacity-building M&E
that result in a plan for evaluating a specific
capacity-building intervention. These steps
are summarized in the checklist that follows.

Checklist: Steps in Designing a

Capacity-Building M&E Plan
The Guide recommends a six-step approach
for developing an M&E plan for capacity
building. The key components of each step
are outlined below.
Define the purpose of the evaluation
(Step 1)
q Evaluators and program planners should
work with key stakeholders to develop
an M&E plan during the design of a ca-

pacity-building or performance improvement intervention.

Capacity-building M&E can be used
internally to improve capacity development interventions or to report results to
external stakeholders. While these two
purposes are not mutually exclusive,
managers should understand the benefits
and drawbacks of emphasizing one objective at the expense of the other.

Define performance objectives (Step 2)

q Capacity is a prerequisite for performance. Evaluators must clearly state the
performance objectives of a capacitybuilding intervention at the outset of
M&E planning and understand the programs approach to improving performance.
q Performance objectives can be expressed
as variables or indicators that can be
measured against international or national standards or locally determined
expectations. Normally, the definition of
performance objectives reflects both external and internal criteria.
Mapping capacity: Build a conceptual
framework for the specific capacitybuilding intervention (Step 3)
q Capacity mapping is a structured process
of thinking through the role capacity
plays in ensuring performance by developing a conceptual framework that is
specific to a particular capacity-building
intervention. Mapping identifies key
factors of capacity and assumptions
about how they interact to influence capacity and performance. If program
planning and M&E design are conducted
simultaneously, capacity mapping can

Summary Checklist: Steps for Designing a Capacity-Building M&E Plan

67

contribute to the choice of intervention

strategies and to the M&E strategy.
The external or operational environment
may have a considerable effect on the
pace, process, outcome, and sustainability of capacity development. It is advisable for program managers to track environmental changes periodically.
Each type of mapping (single-level or
multiple-level) can be done in two or
three iterations. The first iteration of a
map should attempt to provide a full list
of capacity variables that may influence
capacity outcomes and performance. It
should present capacity variables in a
general way. The second or third iteration of a map should be more precise in
depicting the variables to be monitored
over the course of the intervention.
Capacity mapping is sometimes confused with Performance Improvement
(PI). & For clarification, the reader is
referred to the definition of PI in the
Glossary and the table in Annex D.

Identify capacity indicators (Step 4)

q Well-defined indicators provide a reference framework for guiding all
stakeholders toward the same goals. Indicators also allow for standardized
measurement of change during implementation, which enables evaluators to
understand the process of capacity development over time and its relationship
to capacity-building intervention.
q Capacity indicators generally project an
aspiration or a sought-after state or ability. They capture the current stock of
resources available for various uses, or
an individual or organizational behavior
that puts those resources into action
(Moore et al., 2001).
q When selecting capacity indicators it is
advisable to be clear about specific performance and capacity development ob-

68

jectives as well as particular capacitybuilding activities.

Identify appropriate methodological approach and sources of data (Step 5)

q All capacity-building programs need to
be monitored to ensure they are working
well (i.e. to track changes in inputs, processes, outputs and outcomes). However,
the evaluation of program effectiveness
happens less frequently and only for selected interventions due to cost and
complexity.
q Impact evaluation is not advisable in
capacity-building M&E since capacity
measures are not easily quantified, and
identifying similar organizations or systems to facilitate comparison (as in a
case-control study) is difficult.
q Capacity measurement tools should be
able to capture different stages of development of communities, health personnel, organizations, or health systems.
q M&E tools are needed that allow for
repeated measures to capture the interim
steps in capacity-building processes as
well as trends in outcomes.
q Capacity building should be a selfmotivated and self-led process of
change. Evaluation strategies that use
self-assessment techniques and locally
determined benchmarks of progress inspire ownership of capacity development
and increase the likelihood that evaluation results will be used. However, there
can be a cost to this approach in terms of
the perceived validity of findings.
q In the design of capacity-building M&E
strategies, evaluators are advised to use
multiple data-collection instruments,
combine qualitative and quantitative
methods, address more than one level of
capacity and relations between levels,
include self-assessment techniques in
combination with external or standard-

ized methods, triangulate methods and

data sources, and use data interpretation
workshops.

Develop an implementation and dissemination plan (Step 6)

q In disseminating results evaluators
should review findings regularly, and
discuss them with stakeholders to guide
capacity development and encourage
ownership of the M&E process.

Summary Checklist: Steps for Designing a Capacity-Building M&E Plan

69

Annex A

Example of Scoring Used for Measuring Capacity Building in Training, PRIME I

(Fort, 1999)

Dimensions
I. Legal/Policy Support

II. Resources

Objectives
National FP/RH service guidelines
and training are official

Indicator
1. Existence of updated official
FP/RH service and training
guidelines

Political support for training

institutionalization

2. Official (written) policy

supporting institutional training
capacity - e.g., training units,
cadre of master trainers, venues,
etc. - for health providers
3. Favorable public statements on
FP/RH training (for the
improvement of services) at least
twice a year by senior officials
4. </= 20% of training budget
comes from external assistance

Financial
Existence of sufficient and
diversified training budget

Venues/Equipment
Adequate venues

5. Budget covers all aspects of

training (including materials and
equipment, travel and per diem by
consultants and staff, venue hire
and maintenance, etc.)
6. Accessible and available (own,
rented) venues (at least one local
venue in each training area) of
standard quality (continuous
power, food, lighting, acoustics,
and sufficient capacity), accessible
to participants, and available when
needed

Example of Scoring Used for Measuring Capacity Building in Training, PRIME 1

Scoring
0=Nonexistent guidelines (both service
and training), to
4=Complete/updated, disseminated,
and official guidelines
0=Nonexistent written policy to
4=Written/updated, disseminated, and
official
0=No mention, to
4=Mentioned on several private and at
least twice on public occasions
0=No in-country training budgets;
funds are allocated on ad hoc basis, to
4=20% or more of training budget
comes from external assistance
0=Budget does not cover all aspects of
training, to
4=Budget covers all training costs
0=Nonexistent venue, (incrementally
scoring coverage, capacity, and/or
quality of venue), to
4=Fully accessible, high-quality, and
sufficient-capacity local venue for
training events

71

Dimensions

III. Training Plans and

Curriculum

72

Objectives
Materials, equipment, and
supplies (MES)
Appropriate and cost-efficient
MES, (including AV equipment
and teaching aids)
Systems are in place for
replacement and upgrading of
MES

Indicator
7. MES are pertinent, updated,
sufficient, and adapted to local
culture (including locally
produced)

Human
Trainers/preceptors formed have
updated and standardized
technical and presentation
knowledge and skills

9. Trainers/preceptors are
constantly formed (TOT) and do
periodic refresher courses and
pass standard tests on FP/RH
technical and presentation
knowledge and skills

Updated and periodically

reviewed training plan

10. Training plan exists and is

reviewed annually

Updated curriculum is official

standard for training institutions

11. Existence of a standard official

training curriculum guiding
training institutions

8. Financial, printing and planning

capabilities exist for replacing and
upgrading MES

Scoring
0=MES are insufficient and/or
outdated, to
4=MES of standard technical and
material quality and readability are
available for each event participant
0=There are no or insufficient means
for replacing MES, to
4=The means exist to produce, replace
and upgrade MES
0=Trainers/preceptors not regularly
formed and/or do not update their
technical and presentation knowledge
and skills, to
4=Trainers/preceptors constantly
formed and undergoing periodic (at
least once every two years) refresher
courses
0=No training plan performance
(training conducted on ad hoc basis), to
4=Training plans are drawn
periodically (at least annually) and
reviewed
0=No standard training curriculum or
curriculum is inadequate / outdated,
different ones used by different
institutions, to
4=There is a standard curriculum,
reviewed periodically (at least once
every 2 years) and used officially by
training institutions

Dimensions
IV. Organization

Objectives
Leadership
Vision of training as a means to
improve services

Indicator
12. Training plans are linked with
quality of care and increased
service access

Training is an integral part of

organizations strategic planning

13. A training plan and activities

are part of the organizations
strategic plans

Promotion of public-private
collaboration

14. Evidence of public-private

collaboration

Infrastructure
Existence of decentralized training
units in all areas

15. Active training units exist at

central and peripheral levels

Human resource development

HR training (TOT, formative and
refresher courses) is an integrated
part of a Performance
Improvement system (e.g.,
incentives, follow-up and
supervision, efficacy)
Administrative
Existence of a reporting system
for tracking number and
characteristics of trainees and
materials, according to needs

16. HR development is part of a

performance improvement (PI)
strategy

Scoring
0=Providers training plans are not
coupled with service and quality of
care objectives, to
4=Training plans form part of the
quality of care and service
improvement strategies
0=Training is not part of the
organizations strategic plan, to
4=Training is part of the organizations
long-term strategic plan (multiannual)
0=No evidence of public-private
collaboration, to
4=Evidence of public-private
collaboration
0=No decentralized training units (even
if there is one at central level), to
4= Active training units in central and
peripheral levels
0=Training is not coupled with
providers improvement objectives, to
4=Training is part of HR development
and performance

17. Existence and use of a

Training Needs Assessment

0=No TNA customarily done, to

4=TNA is integral and continuous part
of training strategy

18. Existence of an MIS for

trainees and materials matching
TNA

0=No MIS for tracking progress, to

4=MIS for training

Example of Scoring Used for Measuring Capacity Building in Training, PRIME 1

73

Dimensions

V. Community Development
-Participation

74

Objectives
Technical capability
Technological transfer and
development through networking,
evaluation, and research

Indicator
19. Contacts with other training
institutions and institution
evaluation and research feed into
training improvement (e.g., trainee
selection, training contents and
formats)

Track record
Proven capacity to
conduct/replicate courses
autonomously

20. Replica/other courses carried

out independently (with
institutional resources)

Community representatives are

involved in planning and
execution of training activities, are
aware of their rights, and/or
demand competent provider
performance

21. Evidence of community

involvement in providers training
and/or performance assessment
(e.g., quality of care circles)

Scoring
0=No/little use of evaluation and
research of information from other
training institutions to improve, update
training capabilities, to
4=Extensive use of internal and
external data and resources for
improvement
0=No replica or independent courses
carried out by the organization (or only
done with foreign assistance), to
4=Evidence of ongoing
replication/expansion of courses with
institutional resources
0=No/little community involvement, to
4=Extensive involvement /
participation in provider training and/or
performance assessment; organized
demand/petitions to improve services,
etc.

Annex B

Example of Results of PRIME Training Capacity Index (Catotti, 1999)

El Salvador Capacity Building

100.0
56.7

Score

35.8

1997

10.0

1999

2.0

3.7 3.7

3.7

3.0

2.8
2.2 2.2

2.2
1.7

1.51.5 1.7

3.2
2.8 2.8

2.8

1.8

3.5

4.0
2.3

1.7

3.7
2.3
1.7

2.0 2.0

2.5

1.5

3.2

3.3
2.5
1.8

1.0

1.0

1.0
1

10

11

12

13

2.0

2.0

1.3

14

15

16

17

18

1.0
19

1.0
20

21

1997-99 Scores for each of the 20 Indicators and Average Score (21) - Logarithmic Scale

Note: See Annex A for definitions of indicators.

Example of Results of PRIME Training Capacity Index

75

Annex C

Key Internet Resources for Monitoring and Evaluating

Capacity-Building Interventions

There is a wealth of information on capacity measurement and evaluation in general on the

Internet. The list that follows describes Internet sites that focus specifically on capacity measurement; it also includes sites that provide general evaluation information and resources. The
details of many of the capacity measurement tools found on these sites are found in Table 7 in
Part 3 of the Guide. Please note that inclusion on the list does not imply any judgment about any
item listed or not listed.

Capacity Measurement Sites

1. The Managers Electronic Resource Center Management Sciences for Health
http://erc.msh.org/
http://www.msh.org/
The Health Manager's Toolkit is an electronic compendium of tools designed to assist health
professionals at all levels of an organization to provide accessible, high-quality, and sustainable
health services. It is particularly useful for managers who lead others to produce results.
The Health Managers Toolkit includes spreadsheet templates, forms for gathering and analyzing
data, checklists, guidelines for improving organizational performance, and self-assessment tools
that allow managers to evaluate the systems underlying their entire organization. The tools have
been developed by organizations working throughout the world to improve delivery of health
services.
For more information, contact Gail Price or Amanda Ip by e-mail ([email protected]).

2. INTRAH/Prime II
http://www.prime2.org/
The PRIME II Project is a partnership combining leading global health care organizations dedicated to improving the quality and accessibility of family planning and reproductive health care
services throughout the world. Funded by USAID and implemented by the University of North
Carolina at Chapel Hill School of Medicine, PRIME II focuses on strengthening the performance
of primary care providers as they work to improve services in their communities. To accomplish
its goals, PRIME II applies innovative training and learning and performance improvement approaches in collaboration with host-country colleagues to support national reproductive health
goals and priorities.

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions

77

Since 1997, The PRIME Project has been committed to applying the guiding principles of performance improvement (PI) to real-world reproductive health contexts. Work in Yemen, Burkina
Faso, the Dominican Republic, and India indicates that PI users like the clear, highly participatory process and the focus on cost-effective interventions to address the most important problem
areas.
This interactive Website, created by the PRIME II Project and INTRAH, presents a revised edition of Performance Improvement Stages, Steps and Tools, first issued in print form in 2000.
INTRAH/PRIME II published this site online in August 2002 (www.intrah.org/sst/).
For more information, please contact Marc Luoma by email ([email protected]).

3. JHPIEGO
http://www.jhpiego.org
Through advocacy, education and performance improvement, JHPIEGO helps host-country policymakers, educators and trainers increase access and reduce barriers to quality health services,
especially family planning and maternal and neonatal care, for all members of their society.
JHPIEGOs work is carried out in an environment that recognizes individual contributions and
encourages innovative and practical solutions to meet identified needs in low-resource settings
throughout Africa, Asia, and Latin American and the Caribbean.
TIMS is a computer-based tool to track and monitor training efforts. Each persons skills, qualifications, and location are stored, along with courses taken and taught, through a Microsoft Access 2000 database application that stores information about training course content, timing, participants, and trainers. In the standard form, TIMS tracks the following training results over a
period of time:
- Which providers from which service sites have been trained, and in what topic(s)
- Which trainers have been conducting courses, and how many people they have trained
- How many courses have been held, summarized by training center, district, or province
TIMS allows senior and mid-level program managers to monitor the variety of training activities
and track results in a number of perspectives. TIMS is designed to be part of a countrys training
information system, replacing paper-based reporting and aggregation with a computer database.
Ministries of Health, Planning and/or Finance can use TIMS to supplement service information
for policy decisions on training, retraining, and provider deployment.
For additional information about TIMS, contact Catherine Schenck-Yglesias by e-mail
([email protected]).

78

4. Child Survival Technical Support Program (CSTS)

http://www.childsurvival.com/
The Child Survival Technical Support Project (CSTS) assists PVOs funded through the Office of
Private and Voluntary Cooperation's Child Survival Grants Program. The technical support
CSTS provides to PVOs is targeted specifically towards increasing their capacity to achieve
sustainable service delivery in public health interventions.
The programs goal is to help these organizations grow and to develop successful programs that
will continue to serve mothers, children, and communities even when the PVO is no longer present in the area.

5. International Development Research Centre-Canada (IDRC)

http://www.idrc.ca/
The International Development Research Centre (IDRC) is a public corporation created in 1970
to help developing countries find long-term solutions to the social, economic, and environmental
problems they face. IDRCs Evaluation Unit has been working in the area of organizational assessment for over 5 years and has developed a number of tools, including: Enhancing Organizational Performance, a guidebook that presents an innovative and thoroughly tested model for
organizational self-assessment. The tools and tips presented in the guidebook go beyond measuring the impact of programs, products, and services to integrate techniques of formative assessment, in which the assessment team becomes involved in helping its organization become
more effective in meeting its goals. The tools and techniques are flexible, and the model can be
adapted to any type or size of organization. Worksheets and hands-on exercises are included.
Enhancing Organizational Performance will be useful to any organization that is initiating a
process of self-assessment, internal change, or strategic planning. It will appeal particularly to
heads and staff of research organizations, university administrators, staff of research-granting
agencies, and academics and professionals in organizational development and evaluation.

6. International Institute for Sustainable Development (IISD)

http://iisd1.iisd.ca/measure/
IISD has been working on measurements and indicators since 1995, with the aim of making significant local, national, and international contributions, and building the Institute into a world
center of expertise in this field. One of IISDs strategic objectives is to develop robust sets of
indicators for public and private sector decision-makers to measure progress toward sustainable
development and to build an international consensus to promote their use.

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions

79

7. World Health Organization (WHO)

http://www.who.int/whr2001/2001/archives/2000/en/index.htm
World Health Report 2000. Health Systems: Improving Performance
The World Health Report 2000 aims to stimulate a vigorous debate about better ways of measuring health system performance and thus finding a successful new direction for health systems
to follow. By shedding new light on what makes health systems behave in certain ways, WHO
also hopes to help policymakers weigh the many complex issues involved, examine their options,
and make wise choices.

8. USAID Development Experience Clearinghouse (DEC)

http://www.dec.org/
The DEC includes Evaluation Publications such as the TIPS series, which provides guidance on
using the Results Framework, measuring institutional capacity and general quality of indicators
and performance measures.

9. Pact
http://www.pactworld.org/services/oca/index_oca.htm
http://www.pactworld.org/
Pacts unique methodology for organizational capacity assessment and strengthening (OCA)
helps organizations anticipate and overcome the greatest barriers to organizational change and
growth. Through a guided self-assessment and planning process, organizations reflect upon their
performance and select the tools and strategies they need to build capacity and broaden impact.
Pact's OCA is the product of ten years of research and field practice in partnership with the Education Development Center and USAIDs Office of Private & Voluntary Cooperation. Hundreds
of local and international NGOs, private-sector corporations, and municipal governments around
the world have used this methodology.
OCA is a four-staged process that includes:

80

Participatory tool design that empowers organizations to define the critical factors that
influence their performance and to identify relevant indicators for evaluating their competency.

Guided self-assessment that leads employees, board members, and constituents through
structured discussions followed by individual scoring on a series of rigorous performance
indicators.

Data-guided action planning that provides organizations with an opportunity to interpret

the self-assessment data and set change strategies most appropriate to their environment.

Reassessment for continual learning that allows organizations to monitor change, track
the effectiveness of their capacity-building efforts, and integrate new learning as their
needs change and capabilities increase.

For more information on Pacts Organizational Assessment, please contact Betsy Kummer by
email ([email protected]).

Publications Available from Pact

www.pactpublications.org
From the Roots Up: Strengthening Organizational Capacity through Guided Self-Assessment
by World Neighbors
Publisher: World Neighbors
Year: 2000
Basic Guide to Evaluation for Development Workers
by Frances Rubin
Publisher: Oxfam
ISBN: 0-85598-275-6
Year: 1995
This book will help groups to plan for and carry out evaluations as an integral part of development activities. Easy to follow, it focuses on the principles underlying evaluation and deals
clearly and simply with the issues to be considered at the planning stage. It then examines the
steps involved in carrying out different types of evaluation, for specific purposes. The importance of involving local people in evaluations is emphasized throughout.
Participatory Monitoring, Evaluation and Reporting: An Organisational Development Perspective for South African NGOs
by Pact
Publisher: Pact Publications
Year: 1998
This manual explains why participation is important and how to achieve effective stakeholder
participation; the role of monitoring in sustaining progress toward better organizational effectiveness; how evaluation helps an organization to assess its capacity; and the critical role of reporting to stakeholders. It then deals with applying the Organizational Capacity Assessment Tool
(OCAT) in practice, together with examples. A step-by-step guide to designing and implementing a Participatory Monitoring, Evaluation and Reporting (PME&R) information system is included. Although it has been specifically adapted for use by South African NGOs, NGOs can use
OCAT in other countries.

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions

81

10. The International HIV/AIDS Alliance

www.aidsalliance.org/ngosupport
The AIDS Alliance has developed an HIV/AIDS NGO/CBO Support Toolkit that is available on
their Website or by CD-Rom with over 500 downloadable resources and supporting information.
The toolkit includes practical information, tools and example documents to help those working to
establish or improve NGO/CBO support programs. The toolkit also describes key components of
NGO/CBO support programming, based on the Alliance's experience. It also includes resources
from a wide range of other organizations to bring different perspectives and experiences together.
The HIV/AIDS NGO/CBO Support toolkit has been developed for those wishing to establish or
improve NGO/CBO support programs. The toolkit will be useful both for NGO-led support programs and for government-led or multi-sectoral programs, especially in the context of Global
Fund and World Bank financing for NGOs and CBOs working on AIDS. The toolkit will also be
useful to organizations that provide only funding or only training.
Order single or bulk copies of the CD-ROM and supporting publication free of charge from:
[email protected]

11. International NGO Training and Research Centre (INTRAC)

http://www.intrac.org/
International NGO Training and Research Centre (INTRAC) provides support to organizations
involved in international development. Their goal is to improve the performance of NGOs by
exploring relevant policy issues and by strengthening NGO management and organizational effectiveness.
Documents can be ordered through their Website including:
Practical Guidelines for the Monitoring and Evaluation of Capacity-Building: Experiences from
Africa
ISBN: 1 897748-64-7
OPS No. 36, November 2001.
Capacity building and monitoring and evaluation have become two of the most important priorities of the development community during the last decade. Yet they have tended to operate in
relative isolation from each other. In particular, capacity-building programs have been consistently weak in monitoring the impact of their work. This publication aims to help NGOs and donors involved in capacity building to develop appropriate, cost-effective and practical systems
for monitoring and evaluation. While not under-estimating the complexity of these tasks, this
publication puts forward some practical guidelines for designing monitoring and evaluation systems based on experiences with three organizations in different parts of Africa.

82

12. Performance Improvement in Healthcare

http://www.picg.net/
This Website is designed to provide information, tools, and guidelines for planning, implementing, monitoring and evaluating performance improvement processes and activities in health
services delivery organizations. The site is especially tailored for managers, leaders, providers
and other employees working in international health organizations and institutions, whether they
are health ministries or health departments in the public sector or NGOs in the private non-profit
sectors. The site is also for those working as partners with people in these institutions.
Performance Improvement (PI) is a process for enhancing employee and organizational performance that employs an explicit set of methods and strategies. Results are achieved through a systematic process that considers the institutional context; describes desired performance; identifies
gaps between desired and actual performance; identifies root causes; selects, designs and implements interventions to fix the root causes; and measures changes in performance. PI is a continuously evolving process that uses the results of monitoring and feedback to determine whether
progress has been made and to plan and implement additional appropriate changes.
The goal of PI is to solve performance problems or realize performance opportunities at the organizational, process or systems and employee levels in order to achieve desired organizational
results. The overall desired result in our field is the provision of high quality, sustainable health
services.
The Website includes information on the performance improvement process and factors affecting
worker performance, PI tools, and experiences using PI in different health care settings,
For more information or questions email [email protected].

13. Capacity.org
http://www.capacity.org/index_en.html
Capacity.org is a Website dedicated to advancing the policy and practice of capacity building in
international development cooperation. Issue 14 of the web-based magazine Capacity.org presents highlights of the UNDP initiative on capacity building and related information on the policy
and practice of capacity building in international development cooperation (also see UNDP website at http://www.undp.org/dpa/publications/capacity.html).

14. ISNAR/CGIAR - Evaluating Capacity Development in Research & Development Organizations:

http://www.isnar.cgiar.org/ecd/index.htm
This site promotes the use of evaluation as a tool to advance the development of organizational
capacity and performance. Its main purpose is to support a group of managers and evaluators

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions

83

who are evaluating capacity development efforts in their own organizations in Africa, Asia and
Latin America. This site presents the work of a global project, "Evaluating Capacity Development Project (The ECD Project)." National and international research and development organizations are participating in the ECD Project, which is supported by five donor agencies and coordinated by ISNAR.
The site features the ECD Project's activities since 2000 and its result to date. It provides access
to project reports and events. Lists of useful concepts and terms, bibliographic references and
Internet resources are also provided for use by capacity developers and evaluators

15. Reflect-Learn.org - The Organizational Self-Reflection (OSR) Project

http://www.reflect-learn.org/
The Organizational Self-Reflection (OSR) project aims to improve organizational learning by
increasing access to self-reflection tools. The process of reflection implies an organizational diagnosis that will allow learning from experiences, styles of work and results in order to foster
strategic vision, decision making, organizational change and capacity building. The organization
keeps control over orientation of the process and use of results.
The project links a direct service, based on the Internet, and a research agenda designed to create
knowledge about self-reflection and its contribution to organizational learning. The OSR project
seeks to engage diverse organizations in the use of self-reflection resources and also catalyzes
the development of a learning community that focuses on OSR, organizational learning, and the
use of the Internet for institutional strengthening. Several useful frameworks and tools for organizational assessment are presented

16. UNDP United Nations Development Project

http://www.undp.org/dpa/publications/capacity.html
Developing Capacity through Technical Cooperation: Country Experiences provides some concrete inputs to rethinking technical cooperation for todays challenges based on six country
studies Bangladesh, Bolivia, Egypt, Kyrgyz Republic, Philippines and Uganda.
Capacity for Development: New Solutions to Old Problems, with prominent academics and development practitioners as contributors, proposes new approaches to developing lasting indigenous capacities, with a focus on ownership, civic engagement and knowledge. It is a contribution
to a process of debate and dialogue around the broader issue of improving effective capacity development.
Development Policy Journal is a new forum for presenting ideas on applied policies. The subject
of capacity for sustainable development is addressed in this first issue.

84

17. EngenderHealth
http://www.engenderhealth.org
EngenderHealth works worldwide to improve the lives of individuals by making reproductive
health services safe, available, and sustainable. EngenderHealth provides technical assistance,
training, and information, with a focus on practical solutions that improve services where resources are scarce in partnership with governments, institutions, and health care professionals.
EngenderHealth's trademarked COPE (client-oriented, provider-efficient services) is a set of
flexible self-assessment tools that assist providers and supervisors to evaluate and improve the
care offered in clinic and hospital settings. Using self-assessment, client-interviews, client-flow
analysis and facilitated discussion, staff identify areas needing attention and develop their own
solutions and action plans to address the issues. Originally developed for family planning services, COPE has been successfully applied in a variety of healthcare settings all over the world for
over 10 years. With the growing popularity of COPE, healthcare providers from related disciplines asked if the tools could be adapted to a wider range of health services. EngenderHealth
has answered the demand by creating these new products: COPE for Maternal Health Services
and Community COPE: Building Partnership with the Community to Improve Health Services.

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions

85

General Evaluation Sites

1. American Evaluation Association
http://www.eval.org
The American Evaluation Association, an international professional association of evaluators, is
devoted to the application and exploration of program evaluation, personnel evaluation, evaluation technology and other forms of evaluation.
The American Evaluation Association has a Collaborative, Participatory and Empowerment
Evaluation topical interest group that is dedicated to the exploration and refinement of collaborative, participatory and empowerment approaches to evaluation. You can find more information
about them at: http://www.stanford.edu/~davidf/empowermentevaluation.html

2. Canadian Evaluation Association

http://www.evaluationcanada.ca/
The Canadian Evaluation Association is dedicated to the advancement of evaluation for its
members and the public. This site is also available in French.

3. The Evaluation Center at Western Michigan University

http://www.wmich.edu/evalctr/
The Evaluation Center, located at Western Michigan University, is a research and development
unit that provides national and international leadership for advancing the theory and practice of
evaluation, as applied to education and human services.

4. Essentials of Survey Research and Analysis

http://freenet.tlh.fl.us/~polland/qbook.html
This site contains a complete manual entitled Essentials of Survey Research and Analysis: A
Workbook for Community Researchers, written by Ronald Jay Polland, Ph.D.,1998.

5. German Center for Evaluation (in German)

http://www.uni-koeln.de/ew-fak/Wiso/
This is the homepage for the German Center for Evaluation at the University of Cologne. It includes the German translation of the Program Evaluation Standards of the American Evaluation
Society.

86

6. Government Performance Information Consultants

http://members.rogers.com/gpic/evalwebindex.htm
This site offers links to many Web resources on evaluation.

7. The Michigan Association for Evaluation

http://www.maeeval.org/
The Evaluation Promotion Committee has compiled a list of resources in an effort to provide
MAE members and others interested in evaluation with sources for educational materials, tools,
and other resources that may be interesting and helpful. For each resource, the site provides a
brief description (generally from the resource itself) and where to find it.

8. Innovation Network, Inc. (InnoNet)

http://www.innonet.org/
Innovation Network, Inc. (InnoNet) is an Innovation Network, a national nonprofit dedicated to
building the evaluation capacity of nonprofits so they can better serve their communities. InnoNet has two services to meet this end: a search service to find model programs, and an evaluation service that guides agencies through a planning and evaluation process. Description of their
evaluation methodologies and documents available for ordering are listed on this site.

9. International & Cross-Cultural Evaluation Topical Interest Group (I&CCE)

http://home.wmis.net/~russon/icce/
International & Cross-Cultural Evaluation Topical Interest Group is an organization affiliated
with the American Evaluation Association. The purpose of the I&CCE is to provide evaluation
professionals who are interested in cross-cultural issues with an opportunity to share their experiences with one another.

10. MandE News

http://www.mande.co.uk/
MandE News is a news service focusing on developments in monitoring and evaluation methods
relevant to development projects and programs with social development objectives. It is edited
by Rick Davies in Cambridge, UK who can be contacted by email ([email protected]).

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions

87

11. Sociometrics
http://www.socio.com/eval.htm
Sociometrics offers a wide variety of evaluation products and services to professionals across the
world. Their evaluation workshops and training services, technical publications, evaluation tools,
and data sets are all designed to assist practitioners, administrators, evaluators, and funders of
social interventions to design and implement successful evaluation systems.
For additional information, contact Dr. Shobana Raghupathy by email ([email protected]) or
by phone at 1.800.846.3475 x209.

12. Bill Trochim, Cornell University

http://trochim.human.cornell.edu/kb/conmap.htm
Bill Trochim is a faculty member at Cornell University; his work in applied social research and
evaluation is described on this site. His published and unpublished papers, detailed examples of
current research projects, useful tools for researchers, an extensive online textbook, a bulletin
board for discussions and links to other websites related to applied social research methods are
included. Concept mapping is a general method that can be used to help individuals or groups to
describe their ideas about some topic in a pictorial form.

13. UNICEF
http://www.unicef.org/reseval/
This site lists some of the monitoring and evaluation tools recently developed by UNICEF and
its partners, including the UNICEF Guide to Monitoring and Evaluation.

14. United Way

http://www.unitedway.org/outcomes/
The United Ways Resource Network on Outcome Measurement offers a guide to resources for
measuring program outcomes for health, human service and youth- and family-serving agencies.
Their manual, Measuring Program Outcomes: A Practical Approach, can be ordered at the
Website.

88

15. National Science Foundation, Division of Research, Evaluation and Communication

(REC)
http://www.nsf.gov/pubsys/ods/getpub.cfm?nsf97153
This site contains a complete manual, User-Friendly Handbook for Mixed Method Evaluations
(August 1997), edited by Joy Frechtling and Laurie Sharp Westat, and developed with support
from the National Science Foundation, Division of Research, Evaluation and Communication.

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions

89

Annex D

Capacity Mapping and Performance Improvement Compared

Capacity Mapping

Performance Improvement in RH

What is it?

Tool for M&E planning (primarily)

Tool for improving RH services

What is the purpose?

Helps planners and evaluators decide: What M&E approach to

take to determine whether this strategy succeeded in building
capacity (primary use)? What capacity-building strategy to
use? (secondary use).

Helps managers decide: what PI strategy to use? Did performance change as a result of the PI process?

Answers the question

What factors of capacity are required for performance? How

should I measure these factors?

Is progress being made toward goals? Are appropriate actions

being undertaken to promote achieving those goals? What are
the problem areas?

What is the approach?

Conceptual: Evaluators are encouraged to consider a wide

range of factors that might influence capacity and performance.

Focused: Root causes of performance problems are linked to

six performance factors - job expectations; performance feedback; workspace, equipment, and supplies; incentives; organizational support; and knowledge and skills.
Guides organizations in viewing problems systematically and
addressing all areas that enhance performance.
Encourages understanding of the organization as a system of
interdependent functions and people.

Guides planners and evaluators in viewing capacity systematically and identifying all areas that affect performance.
Encourages understanding of capacity in the health sector as a
system that includes four interdependent levels: the system,
organizations, health personnel, individuals and communities.
When to use it?

Can be used to organize and analyze information before or

after a capacity-building intervention is designed.

Used to organize and analyze information before deciding what

intervention is needed.

Focus of study/action

Applies to systems, organizations, humans, and communities

Applies to humans within organizational systems

Who is involved?

Encourages stakeholder involvement

Encourages stakeholder involvement

View of performance

Performance is the result of capacity and context

Human performance is a factor of knowledge, skills, capacity

and motives, and context

Capacity Mapping and Performance Improvement Compared

91

Glossary
Capacity is the ability to carry out stated objectives. It has also been described as the stock of
resources available to an organization or system as well as the actions that transform those resources into performance.
Capacity building (or capacity development) is a process that improves the ability of
a person, group, organization, or system to meet objectives or to perform better.
Capacity evaluation is normally more complex than monitoring, and is conducted to gain understanding of the relationship between capacity-building interventions and capacity outcomes,
or the links between capacity and performance variables.
Capacity mapping is a structured process of thinking through the role capacity plays in ensuring
performance by developing a conceptual framework that is specific to a particular capacitybuilding intervention. During capacity mapping, all the possible factors of capacity that influence
performance and the relationships between them must be identified. Once the factors are all laid
out, the program staff or evaluator can focus on those that are most essential for the evaluation.
Capacity monitoring normally would be used to understand the effectiveness and efficiency of
a capacity-building intervention during implementation (i.e., is capacity improving and at what
cost?) to contribute to strategic or operational decisions related to capacity building or enable a
periodic look at a program or system.
Cold chain: The system that ensures vaccine viability from manufacturing to delivery.
Contextual factors: external factors relating to the economic, social, cultural and political environment. Factors normally outside the control of most health sector actors.
Impact: Long-term results achieved through improved performance of the health system: sustainable health system and improved health status. Impact measures are not addressed in capacity-building M&E.
Impact evaluation: An evaluation that uses experimental or quasi-experimental study design to
attribute changes in capacity or performance to program interventions. Impact evaluation is not
appropriate or useful in the context of capacity-building M&E because of the difficulty of quantifying many elements of capacity and attributing capacity change to any single intervention or
even a range of them.
Input: Set of resources, including service personnel, financial resources, space, policy orientation, and program service recipients, that are the raw materials that contribute to capacity at each
level (system, organization, health personnel, and individual/community).
Outcome: Set of results that represent capacity (an ability to carry out stated objectives), often
expected to change as a direct result of capacity-building intervention.

Glossary

93

Output: Set of products anticipated through the execution of practices, activities, or functions.
Performance: Set of results that represent productivity and competence related to an established
objective, goal or standard. The four capacity levels together contribute to overall system-level
performance.
Performance Improvement (PI): Performance Improvement (PI) is a process for enhancing
employee and organizational performance that employs an explicit set of methods and strategies.
Results are achieved through a systematic process that considers the institutional context; describes desired performance; identifies gaps between desired and actual performance; identifies
root causes; selects, designs and implements interventions to fix the root causes; and measures
changes in performance. PI is a continuously evolving process that uses the results of monitoring
and feedback to determine whether progress has been made and to plan and implement additional
appropriate changes.
Process: Set of activities, practices, or functions by which the resources are used in pursuit of the
expected results.
Theory of action: Part of a capacity-building plan that includes common objectives and shared
concepts. A coherent theory of action agreed on by the key groups involved in the process states
how activities are expected to produce intermediate and longer-term results and benefits. Without a theory of action, a capacity development effort could become a fragmented exercise in
wishful thinking, rather than a coherent initiative with a high probability of success (Horton,
2001).
Triangulation: The use of multiple data sources or methods to validate findings, discover errors
or inconsistencies, and reduce bias.

94

Bibliography
Africa Bureau, Office of Sustainable Development USAID. 1999. Health and Family Planning
Indicators: Measuring Sustainability, Volume II. Washington: USAID.
Ampomah, K. 2000. PRIMEs Technical Report 20: An Assessment of the Impact of PRIMEs
Interventions on the Training Capacity and Reproductive Health Service Delivery in Ghana.
2000. Chapel Hill, NC: INTRAH.
Bertrand, J. and Escudaro, G. 2002. Compendium of Indicators for Evaluating Reproductive
Health Programs, Volume 1. Chapel Hill: MEASURE Evaluation Project.
Brown, L., LaFond, A., Macintyre, K. 2001. Measuring Capacity Building. Chapel Hill:
MEASURE Evaluation Project.
Catotti, D. 1999. PRIMEs Technical Report 13: Improving the Quality and Availability of Family Planning and Reproductive Health Services at the Primary Care Level: Institutional Capacity
Building in the El Salvador Ministry of Health. Chapel Hill: INTRAH.
Development Resources Team, World Vision. 2002. Transformational Development Indicators
Field Guide. Washington, DC: World Vision.
Earl, S., Carden, F., and Smutylo, T. 2001. Outcome Mapping: Building Learning and Reflection
into Development Programs. Ottawa: International Development Research Centre.
Eng, E. and Parker, E. 1994. Measuring Community Competence in the Mississippi Delta: The
Interface between Program Evaluation and Empowerment. Health Education Quarterly 21 (2):
199-220.
Figueroa, M.E., Kincaid D.L., Pani, M. and Lewis, G. 2002. Communication for Social Change:
An Integrated Model for Measuring the Process and Its Outcomes. Communication for Social
Change Working Paper Series, No. 1. Baltimore: Johns Hopkins Center for Communications
Programs.
Fort, Alfredo. 1999. PRIMEs Technical Report 16: Capacity Building in Training: A Framework and Tool for Measuring Progress. Chapel Hill: INTRAH.
Franco, L.M., Bennett, S. and Kanfer, R. 2002. Health Sector Reform and Public Sector Health
Worker Motivation: A Conceptual Framework. Social Science and Medicine 54: 1255-1266.
Goodman, R.M., Speers, M.A., McLeroy, K., Fawcett, S., Kegler, M., Parker, E., et al. 1998.
Identifying and Defining the Dimensions of Community Capacity to Provide a Basis for Measurement. Health Educ Behav 25 (3): 258-278.
Gubbles, P., Koss C. 2000. From the Roots Up: Strengthening Organizational Capacity through
Guided Self-Assessment. Oklahoma City: World Neighbors.
Bibliography

95

Horton, D. 2002. Capacity Development in Planning, Monitoring, and Evaluation: Results of an

Evaluation. Briefing Paper No. 51. ISNAR.
Horton, D. 2002. Planning, Implementing and Evaluating Capacity Development. Briefing Paper
No. 50. ISNAR
Horton, D. (ed). 2001. Learning about Capacity Development through Evaluation Perspectives
and Observation from a Collaborative Network of National and International Organization and
Donor Agencies. The Hague: International Service for National Agricultural Research.
Horton, D., Mackay, R., Andersen, A., and Dupleich, L. 2000. Evaluating Capacity Development
in Planning, Monitoring, and Evaluation: A Case from Agricultural Research. Research Report
17. The Hague: International Service for National Agricultural Research.
INTRAC. 1998. The Newsletter of the International NGO Training and Research Center. No. 11.
Israel, B., Schultz, A., Parket, E., and Becker, A. 1998. Review of Community-Based Research:
Assessing Partnerships Approaches to Improve Public Health. Annual Review of Public Health
19:173-202.
Israel, B.A., Checkoway, B., Schulz, A., Zimmerman, M., 1994. Health Education and Community Empowerment: Conceptualizing and Measuring Perceptions of Individual, Organizational,
and Community Control. Health Education Quarterly 21(2): 149-170.
James, R. 2001. Practical Guidelines for the Monitoring and Evaluation of Capacity Building
Experience from Africa. London: Intrac.
Kaul Shah, M., Degnan Kambou, S., Monahan, B. 1999. Embracing Participation in Development: Wisdom from the Field. Atlanta: CARE Health and Population Unit.
Knight, R.J., Tsui, A.O. 1997. Family Planning Sustainability at the Outcome and Program Levels: Constructing Indicators for USAID Strategic Planning. Chapel Hill: The EVALUATION
Project.
Kotellos, K.A., Amon J.J., Githens Benazerga, W.M. 1998. Field Experiences: Measuring Capacity Building Efforts in HIV/AIDS Prevention Programs. AIDSCAP Family Health International, AIDS 12 (suppl. 2): 109- S117.
LaFond, A. 1995. Sustaining Primary Health Care. London: Earthscan.
LaFond, A., Brown, L. and Macintyre, K. 2002. Mapping Capacity in the Health Sector: A Conceptual Framework. International Journal of Health Planning and Management. 17:3-22.

96

Lake, S., Daura, M., and Mabanddhala, M., et al. 2000. Analyzing the Process of Health
Financing Reform in South Africa and Zambia. Zambia Country Reports. Major Applied Research Technical Paper 1. Bethesda: Partnerships for Health Reform Project.
Lande, R.E. 2002. Performance Improvement. Population Reports, Series J, No. 52, Baltimore:
The Johns Hopkins Bloomberg School of Public Health, Population Information Program.
Luoma, M. 2000. PRIMEs Technical Report 19: Dominican Republic Performance Improvement Project Evaluation. Chapel Hill: INTRAH.
Lusthaus, C., Adrien, M., Andersen, G., and Carden, F. 1999. Enhancing Organizational Performance: A Toolbox for Self-Assessment. Ottawa: International Development Research Centre.
Mackay, R. and Horton, D. 2002. Capacity Development in Planning, Monitoring, and Evaluation: Results of an Evaluation. Briefing Paper No. 51. ISNAR.
Management Sciences for Health. 1996. Planning for Sustainability: Assessing the Management
Capabilities of Your Organization. The Family Planning Manager. FPMD.
McCaffrey, J., Luoma, M., Newman, C., Rudy, S., Fort, A., Rosensweig, F. 1999. PI Stages,
Steps and Tools, Chapel Hill: INTRAH.
MEASURE Evaluation. 1998. The Needs Assessment Validation Study and 1998 Institutional
Capacity Assessment, PASCA Project. Chapel Hill: MEASURE Evaluation Project.
MEASURE Evaluation. 2001. Mapping Capacity in the Health Sector: Application of the
MEASURE Conceptual Framework in Measuring Capacity Building for a Complex Nongovernmental Organization. Draft.
Moore, M., Brown, L., and Honan, J. 2001. Toward a Public Value Framework for Accountability and Performance Management for International Non-Governmental Organizations. Presented
at Hauser Center/Keio University Workshop on Accountability for International Nongovernmental Organizations, November 2 11, 2001.
Morgan, P. 1997. The Design and Use of Capacity Development Indicators. CIDA.
Murray, C.J.L., Frenk J. 1999. A WHO Framework for Health System Performance Assessment.
Geneva: World Health Organization.
Oakley, P. 2001. Evaluating Empowerment: Reviewing the Concept and Practice. INTRAC
NGO Management and Policy Series No.13. London: INTRAC.
Partnerships for Health Reform. 1997. Measuring Results of Health Sector Reform for System
Performance: A Handbook of Indicators. Bethesda: Partnerships for Health Reform.

Bibliography

97

Plummer, J. 1999. Municipalities & Community Participation: A Sourcebook for Capacity

Building. London and Sterling, VA: Earthscan Publications Ltd.
PRIME II. 2001. PRIME II Performance Monitoring Plan (PMP): General Guidelines for Monitoring and Reporting on the 10 Key PMP Indicators. Version 1.01. Chapel Hill: INTRAH.
Pyle, D. and LaFond, A. 2001. Case Example: Measuring Capacity Building in Training PRIMEs Evaluation, Documentation and Dissemination (EDD); draft.
Ross, J.A. and Mauldin, W.P. 1996. Family Planning Programs: Efforts and Results, 1972-1974.
Studies in Family Planning 27 (3):137-147.
Sarriot, E. 2002a. The Child Survival Sustainability Assessment (CSSA), For Shared
Sustainability Evaluation Methodology in Child Survival Interventions. Maryland: CORE
CSTS.
Sarriot, Eric. 2002b. Sustaining Child Survival: Many Roads To Choose, but Do We
Have a Map? Maryland: ORC Macro.
Sullivan, T. and Bertrand, J. (eds). 2000. Monitoring Quality of Care in Family Planning by the
Quick Investigation of Quality (QIQ): Country Reports. Technical Report Series, No. 5. Chapel
Hill: MEASURE Evaluation.
UNICEF. 1999. M&E of Capacity Building: Guidance and Tools. Working Draft.
World Health Organization. 2000.World Health Report 2000: Health Systems: Improving Performance. Geneva: World Health Organization.

98