CHAPTER 5

Developing Indicators
 Learning outcomes

At the end of the session, participants will

be able to:
• Critique indicators
• Identify criteria for selection of sound indicators
• Understand how indicators are linked to the
frameworks
• Select indicators and complete an Indicator
Reference Sheet
 What is an indicator?

• An indicator is a qualitative or
quantitative measure of program
performance that is used to
demonstrate changes and which
details whether the program results
are being or have been achieved.
Purposes of indicators

• Defining objectives
• Assessing present and future direction with
respect to goals and values
• Evaluating specific programmes
• Demonstrating progress
• Measuring changes in a specific condition
or situation over time
• Determining impact of programmes and
conveying messages
Criteria for indicator selection

• Policy-relevant
• Specific
• Valid
• Reliable
• Sensitive
• Measurable
• User-friendly
• Cost-effective
Assessment of data quality

• Relevance • Completeness
• Timeliness • Comparability
• Accessibility • Methodological
• Clarity soundness
• Metadata • Efficiency
• Accuracy • Coherence
 Sustainable Development -
frameworks and indicators

• Frameworks and approaches

• Lists of indicators
• Focus of this presentation is on the
Commission on Sustainable
Development (CSD) list of indicators
and the Millennium Development
Indicators
 Quantitative and qualitative
Quantitative Qualitative
• Number • Compliance with
• Percent • Quality of
• Rate • Extent of
• Ratio/Proportion • Level of
 Input indicator
Input indicators are quantified and
time-bound statements of resources to
be provided. Here are a few examples
of this indicator:
• Vehicle operating costs for the crop
extension service
• Levels of financial contributions from
the govt, or co-financers
• Appointment of staff
• Provision of building
 Performance indicator
Performance indicators measure what happens during
implementation.
• Often they, are tabulated as a set of contracted
completions or milestone events taken from an activity
plan. Examples are
• Date by which building site clearance must be
completed
• Latest date for delivery of fertilizer to farm stores
• Number of health outlets reporting family planning
activity
• Number of women receiving a contraceptive counseling
• Status of procurement of school textbooks
 Output/outcome indicator

• Output indicators show the immediate physical and

financial outputs or outcomes of the Project: physical
quantities, organizational strengthening, and initial flow of
services.
• They include program performance measures based on
cost or operational ratios.
• Examples include
– Several teachers trained in textbook use.
– Cost per kilometer of road construction.
– Crop yield per acre of land.
– The ratio of textbooks to students.
– Time is taken to process a credit application.
 Indicator Pyramid
Decreases

Global Number of
Compare countries
Overview world-wide situation Indicators

Increases
National/Sub-national
Assess effectiveness of response
Reflect goals/objectives of national/sub-national response

District or Facility
Identify progress, problems, and challenges
Results-Based Monitoring:
Adult Literacy
Goal • Higher income levels; increase
(Impacts) access to higher skill jobs

Outcomes • Increased literacy skill; more

employment opportunities

Outputs • Number of adults completing

literacy courses

Activities • Literacy training courses

Inputs • Facilities, trainers, materials

 How do you develop
appropriate indicators?
• Involve your program stakeholders in
indicator development.
• Review evaluation questions and use your
logic model as a template to develop
indicators.
• Review indicators to ensure they are
specific, observable, and measurable
• Include baseline data for inputs and
outcomes if you are trying to measure
change.
 How to Develop an Indicator
Objective Criteria Indicato Baselin Target Data Sources
r e
The child is Body temperature ºC 39º 36-37º Measurement with
healthy fever thermometer
1. Criteria: What criteria do we use for verifying the achievement of the
objective (output, outcome, goal)? What (measurable) specific aspect or
characteristics of the situation described in the objective do we observe and
analyse?
2. Indicator: What do we measure? What is the measuring unit (percentage,
hectare)?
3. Baseline: What is the measuring result at the beginning of the project?
4. Target: What is the target value we want to achieve till the end of the
project? In some case it might be useful to define intermediate targets or
milestones.
5. Source: Where do we obtain the information from? What methods do we use
for collecting data? When and how frequent do we collect data?
 Selecting Outcome
Indicators
A good indicator must be:

Clear (Precise and unambiguous)

Relevant (Appropriate to subject at hand)

Economic (Available at reasonable cost)

Adequate (Must provide a sufficient basis to assess
performance)
Monitorable (Must be amenable/agreeable to
independent validation)

Salvatore-Schiavo-Campo 2000
 Developing A Set of Outcomes

Outcomes Indicators Baselines Targets

1. Fewer children 1. % children
seeking treatment sleeping under
for malaria bed nets

2. Morbidity rates of
malaria in kids
under 15

2. Learning outcomes 1. % of Grade 6

for primary school students scoring
children improve 70% or better on
standardized
math and science
tests
 Result change

Inputs Activities Outputs Objective (outcome Goal

level) (Impact level)

Indicators (SMART indicators)

∙ 3 trainers ∙ Number of For example: ∙ By the end the project, Increase in
∙ 60 trainees, activities 35 out of 56 50% of women in the number of
30 of whom Number of women target areas.. communities
were trainings participants.. ∙ 30% of men in target who
women. ∙ Record of areas helped their demonstrate
∙ Budget plan expenditures wives respect for
women

20
 Checklist for Assessing Proposed
Indicators

Outcome to be measured: ______________________________

Indicator selected: ____________________________________

Is the Indicator…

1 As direct as possible a reflection of the outcome itself? ✔

2 Sufficiently precise to ensure objective measurement?

3 Calling for the most practical, cost-effective collection of data

4 Sensitive to change in the outcome, but relatively unaffected by other

changes?
5 Disaggregated as needed when reporting on the outcome?

United Way of America

 Developing Baseline Data
:
Outcomes Indicators Baselines Targets
1. Fewer children seek 1.% if children 40
treatment for malaria sleeping under bed
nets

2. Morbidity rates of
malaria in children
under 15 years

1.Learing outcomes for 1. %grade six 1.

primary school children students scoring
improve 70% or better
on standardized
math and
science tests
 Developing Targets

Outcomes Indicators Baselines Targets

1. Fewer children seek 1 % of children 1. 40% urban children 1. 85% urban children ages
treatment for . sleeping under ages 3-5 in 2007 3-5 by 2010
malaria bed nets

2 Morbidity rates of 2. 40% rural children in 2. 10% rural children aqes 3-5
. malaria in children 2007 in 2010
under 15 years

2. Learning outcomes 1 % of Grade 6 1. 75% in 2002 scored 1. 80% scoring 70% or better
for primary school . students scoring 70% or better in math. in math by 2006.
children improve 70% or better on
61% in 2002 scored 67% scoring 70% or better
standardized math
70% or better in in science by 2006.
and science tests
science
 Exercise: Group work
• Use your case study and identify at
least five indicators for program
monitoring
 Key Criteria
for Collecting Quality Performance
Data
Reliability

Validity Timeliness
Steps for developing indicators
 Step 1: Identify What to
Measure
• The first step to creating program indicators for monitoring and
evaluation is to determine which characteristics of the program are
most important to track.
• A program will use many indicators to assess different types and
levels of change that result from the intervention, like changes in
certain health knowledge, attitudes, and behaviors among the
priority audience(s).
• Referring to the program’s logic model can help to identify key
program areas that need to be included in monitoring indicators.
• Indicators fall under the three stages of the logic model, which
include:
– Inputs – resources, contributions, and investments that go into a
program
– Outputs – activities, services, events and products that reach the priority
audience(s)
– Outcomes – results or changes for the priority audience(s)
 Step 2: Use the SMART
Process to Develop
High-Quality Indicators
• One way to develop good indicators is to use the SMART criteria, as
explained below. Consider each of these points when developing
new indicators or revising old ones.
• Specific: The indicator should accurately describe what is intended
to be measured, and should not include multiple measurements in
one indicator.
• Measurable: Regardless of who uses the indicator, consistent
results should be obtained and tracked under the same conditions.
• Attainable: Collecting data for the indicator should be simple,
straightforward, and cost-effective.
• Relevant: The indicator should be closely connected with each
respective input, output or outcome.
• Time-bound: The indicator should include a specific time frame.
 Step 3: Establish a Reference Point
Intervention has not begun Intervention has begun Intervention is over
Establish the reference See if any data related to A reference point can be
point immediately before it the program indicators were established through a
begins. This point is usually collected in other surveys control group. Identify a
referred to as a baseline. targeting similar sample group that has not
populations. For example, been exposed to the
use data from large-scale intervention and is
national surveys like DHS demographically,
geographically, cuturally,
and socially similar to the
intervention group. Then
administer data collection
on program indicators with
this group.
If comparable
measurements in other
surveys/programs cannot
be found, use the program
indicators to collect data on
the current state of the
•
Step 4: Set Targets
Targets define the path and end destination of what a program
hopes to achieve and is a number or percentage which will
measure success. Once the reference point is established,
determine what changes should be seen in the program’s
indicators that would reflect progress towards success.
• When establishing targets, consider:
• Baseline data or reference point: This sets a certain point in
time in the program from which to observe change over time.
• Stakeholder’s expectations: Understanding the expectations of
key stakeholders and partners can help set reasonable
expectations for what can be achieved.
• Recent research findings: Do a literature search, if literature is
available, for the latest findings about local conditions and the
program sector, or conduct FGDs or IDIs in order to set
realistic targets.
• Accomplishments of similar programs: Identify relevant
information on similar programs that have been implemented
under comparable conditions. Those with a reputation for high
performance can often provide critical input on setting targets.
 Step 5: Determine the
Frequency of Data Collection
 Data Collection Methods

Panel
Key informant Surveys
interviews
Conversation
with Focus
concerned One-Time
Group
individuals Survey
Interviews
Participant
Community Observation Direct
Census
Interviews observation
Reviews of
official records
(MIS and admin Field
Field data) experiments
visits Questionnaires

Informal/Less Structured Methods More Structured/Formal Methods

 Types of Indicators

Indicators can be classified as follows:

•Quantitative Indicators (Output Indicators)
•Qualitative Indicators (Outcome /
Performance Indicators)
 Quantitative Indicators / Output
Indicators:

• The number of people attending a training

• The weight of fish caught
• Unemployment (By age, gender, Occupation)
• Per Capita Income
• No. of Community Organizations
• Rates of HIV Infection
• The average rice harvest per hectare
• The cost of transport to market
• Increase in household income
• Infant Mortality Rate
 Performance Indicators /
Qualitative Indicators:

• Greater freedom of expression

• Ease of access to a facility
• Participation in Youth Groups
• Participation Levels in Sports
• Increased Hopes of the people
towards betterment of the
democratic systems
• Women’s participation in decision
making
 Comparison of Major
Data Collection Methods
Date Collection Method

Characteristic Review of Self-Administered Interview Rating by Trained

Program Records Questionnaire Observer
Depends on
Cost Low Moderate Moderate to High Availability of
Low-Cost Observers

Amount of
Training Required Some None to Some Moderate to High Moderate to High
for Data
Collectors

Depends on Amount
Completion Time of Data Needed
Moderate Moderate Short to Moderate

High, if Records Depends on How Generally Moderate

Response Rate Contain Needed Data Distributed to Good
High

United Way of America

Examples
 SOCIAL
Theme Sub-theme Indicator
Percent of Population Living below Poverty Line
Poverty Gini Index of Income Inequality
Equity Unemployment Rate
Gender Equality Ratio of Average Female Wage to Male Wage
Nutritional Status Nutritional Status of Children
Mortality Rate Under 5 Years Old
Mortality
Life Expectancy at Birth
Sanitation Percent of Population with Adequate Sewage Disposal Facilities
Health (6) Drinking Water Population with Access to Safe Drinking Water
Percent of Population with Access to Primary Health Care
Facilities
Healthcare Delivery Immunization against Infectious Childhood Diseases
Contraceptive Prevalence Rate
Children Reaching Grade 5 of Primary Education Adult
Education Level
Education (36) Adult Secondary Education Achievement Level
Literacy Adult Literacy Rate

Housing (7) Living Conditions Floor Area per Person

Security Crime (36, 24) Number of Recorded Crimes per 100,000 Population

Population Growth Rate

Population (5) Population Change
Population of Urban Formal and Informal Settlements
 ENVIRONMENTAL
Theme Sub-theme Indicator
Climate Change Emission of Greenhouse Gases
Atmosphere Ozone Layer Depletion Consumption of Ozone Depleting Substances
Air Quality Ambient Concentration of Air Pollutants in Urban Areas
Arable and Permanent Crop Land Area
Agriculture (14) Use of Fertilizers

Use of Agriculture Pesticides

Land (10)
Forest Area as a Percent of Land Area
Forest (11)
Wood Harvesting Intensity

Desertification (12) Land Affected by Desertification

Urbanization (7) Area of Urban Formal and Informal Settlements
Algae Concentration in Coastal Waters
Oceans, Seas and Coastal Zone
Percent of Total Population Living in Coastal Areas
Coasts (17)
Fisheries Annual Catch by Major Species
Annual Withdrawal of Ground and Surface Water as a
Water Quality Percent of Total Available Water
Fresh Water (18) BOD in Water Bodies
Water Quality
Concentration of Faecal Coliform in Freshwater
Areas of Selected Key Ecosystems
Ecosystem
Biodiversity (15) Protected Area as a % of Total Area
Species Abundance of Selected Key Species
 ECONOMIC
Theme Sub-theme Indicator
GDP per Capita
Economic Performance
Investment Share in GDP
Economic
Trade Balance of Trade in Goods and Services
Structure (2) Debt to GNP Ratio
Financial Status (33)
Total ODA Given or Received as a Percent of GNP
Material Consumption Intensity of Material Use
Annual Energy Consumption per Capita

Energy Use Share of Consumption of Renewable Energy Resources

Intensity of Energy Use
Consumption
Generation of Industrial and Municipal Solid Waste
and Production
Patterns (4) Waste Generation and Generation of Hazardous Waste
Management (19-22) Generation of Radioactive Waste

Waste Recycling and Reuse

Transportation Distance Traveled per Capita by Mode of Transport

 Common Problems in
Specifying Indicators
1. The indicators are irrelevant and do not correspond to the output
level;
2. The indicators do not include an objective standard against
which achievement can be assessed;
3. The indicators constructed are without reference to the baseline;
4. The indicators are numerous and redundant with little
consideration of time, human resources and cost required to
collect data for the construction of the indicators;
5. The indicators are unrealistic and sometimes very difficult to
conceptualize and measure;
6. The indicators are not representative of the universe.
Challenges

• Lack of data and development capacity

• International harmonization efforts
• Further indicator development in “less
advanced” substantive areas
• Training in integrated information
management
THE END