Project Monitoring and

Evaluation (PMgt-662)

CHAPTER TWO
Frameworks and Indicators for Project
Instructor: Geremew Kefyalew Monitoring and Evaluation
(Assist. Professor)
Learning Objectives

At the end of the chapter, students will be able to:

1. Understand M&E frameworks and systems

2.Identify and differentiate between conceptual frameworks, results and

logical frameworks,

3.Define goals and objectives for specific intervention programs

4.Design frameworks for specific projects

5.Discuss how frameworks are used for M&E planning

Chapter two contents:

2.0 Introduction

2.1 The Conceptual Framework Approach

2.2 The Logical Framework approach

2.3 Results-oriented approaches

2.4 Understanding indicators

2.5 Selecting indicators and setting targets

2.6 Using comparable and core indicators

 Introduction
What is Framework?
• A framework is a basic conceptual idea around or over which something is built. It
is a system of rules, ideas, or beliefs that is used to plan or decide something.

What is system?

• System is a set of connected items or devices that operate together. A system is

a group of interacting or interrelated elements that act according to a set of rules
to form a unified whole. A system, surrounded and influenced by its environment,
is described by its boundaries, structure and purpose and expressed in its
functioning.
Monitoring and Evaluation (M&E) Framework
• As with most things in international development, there is no standard definition of a Monitoring
and Evaluation (M&E) framework, or how it differs from an M&E plan.

• For many organisations, an M&E framework is a table that describes the indicators that are used
to measure whether the program is a success.

• The M&E framework then becomes one part of the M&E plan, which describes how the whole
M&E system for the program works, including things like who is responsible for it, what forms and
tools will be used, how the data will flow through the organisation, and who will make decisions
using the data. In other organisations the whole M&E plan is called an M&E framework.

• A M&E plan describes how the whole M&E system for the program works which includes the
indicators, who is responsible for collecting them, what forms and tools will be used, and how the
data will flow through the organisation.
Purpose of M&E Frameworks
• A clear framework is essential to guide monitoring and evaluation.

• A framework should explain how the project/programme is supposed to work by laying out the components
of the initiative and the order or the steps needed to achieve the desired results.

• A framework increases understanding of the programme’s goals and objectives, defines the relationships
between factors key to implementation, and articulates the internal and external elements that could affect
the programme’s success.

• Help to develop sound monitoring and evaluation plans and implementation of monitoring and evaluation
activities

• Define relationships among inputs, activities, outputs, outcomes and impacts

• Clarify the relationship between programme activities and external factors.

• Demonstrate how activities will lead to desired outcomes and impacts, especially when resources are not
available to conduct rigorous impact evaluations. They often display relationships graphically.
Considerations when developing a M&E framework
•Asking questions:

 What are the objectives of the monitoring activities?

 What are the specific questions that need to be asked to gauge the progress of the intervention?

 What information is needed to see if activities are being implemented in the way that was planned, and
who can provide that information?

 What are the objectives of the evaluation?

 What are the specific questions that need to be answered to gauge the impact and success of the
intervention?

 What information is needed to determine if the expected objectives and outcomes were accomplished
and who can provide that information?

 Determining whether the questions being asked are appropriate ones for understanding how
“successful” the intervention has been with respect to its expected objectives and outcomes?
 Developing a corresponding monitoring and evaluation plan that acts as a
monitoring tool by defining how information from the programme will be
tracked.

 Developing the framework and plan before activities are implemented.

 Determining which framework is best to use. A number of different

frameworks are used or requested by organizations and donors. Some
donors combine aspects of frameworks in a customized approach. Others
do not include explicit guidance for programmes around the selection of a
framework. Programmes should select the type of framework that best suits
their strategies and activities and responds to institutional requirements.
It is also important to keep in mind that:
 Different kinds of interventions (policy change, awareness raising campaigns, community
mobilization, improving service delivery and response) will need different kinds of frameworks,
tools and indicators.

 An appropriate framework for monitoring and evaluation of activities can be designed and
implemented even when a) programmes do not have significant resources b) programme staff and
implementers, service providers and policy makers feel they do not have additional time to devote
to monitoring and evaluation.

 Many existing tools can be adapted to specific contexts and monitoring and evaluation needs. If
monitoring and evaluation activities and tools are considered and built into programmatic work or
service provision from the start, the resource and time burden is minimized.

 It is important to clarify objectives, what information will be most useful in reaching those objectives
and what information is already available or easily collected.
 Steps in Developing M&E Frameworks
•Programme implementers (often from diverse sectors) should jointly take steps in developing the
monitoring and evaluation framework:

1) They should determine the purposes of the monitoring and evaluation mechanisms and assess the
information needs of each actor.

2) Ensure prevention and response interventions have clearly defined

objectives, outputs and indicators;

3) Establish coordinated and common reporting tools;

4) Determine methods for obtaining information on indicators;

5) Assign responsibilities for information gathering, determine time frame and frequency of data
collection, and allocate resources; and

6) Establish mechanisms for sharing information and incorporating results into prevention and
response planning. (UNHCR Guidelines, 2003)
2.1. Alternative M&E Frameworks
• Monitoring and Evaluation (M&E) is an essential component of any intervention, project, or program.

• Frameworks are key elements of M&E plans that depict the components of a project and the sequence
of steps needed to achieve the desired outcomes.

• They help increase understanding of the program’s goals and objectives, define the relationships
between factors key to implementation, and delineate the internal and external elements that could
affect its success.

• They are crucial for understanding and analyzing how a program is supposed to work.

• There is no one perfect framework and no single framework is appropriate for all situations, but several
common types will be discussed here:

1. Conceptual framework

2. Results framework
 1. Conceptual Frameworks (aka. research or theoretical frameworks)
What is a conceptual framework (CF)?
 CF is a diagram that identifies and illustrates the relationships between all relevant systemic,
organizational, individual, or other salient factors that may influence program/project operation and
the successful achievement of program or project goals. It is useful for identifying and illustrating
the factors and relationships that influence the outcome of a program or intervention.
 Conceptual frameworks are also known as theoretical or causal frameworks. They inform healthy
program design by making explicit connections between relevant contextual factors and the
purpose of the program; clarify the “why” and “how” questions of program operation and design—
that is, the causal links between program activities and objectives while defining the underlying
assumptions made to achieve program goals; and guide the selection of appropriate indicators.
 Conceptual frameworks need to be as comprehensive as possible. Do not limit it to just what you
will do, include other elements that will affect your program so that you will understand these
relationships and be able to incorporate other elements if necessary.
The purpose of CF:
To show where program fits into wider context;

To clarify assumptions about causal relationships;

To show how program components will operate to influence outcomes;

To guide identification of indicators;

To guide impact analysis (causal pathways).

Conceptual Frameworks - Example
Individual
characteristics
Healthy Health
Service practices status
utilization

Institutional Program
Program supply
capacity sustainability

Technical
inputs
• This is a very general example of a conceptual framework in the health field. It has the characteristics described in the previous slide that it includes

a number of factors that affect the desired outcome, including those not under the control of the program (such as individual factors). In each box is

a factor related to another factor that ultimately affect a health outcome. Arrows indicate the relationship between factors, which in this case include

individual, systemic and program variables.

• As you look at the framework, notice that there are two outcomes: health status and program sustainability. Following health status backward, we

see that healthy practices influence health status. Healthy practices are affected by individual characteristics, service utilization and institutional

capacity. You can also see service utilization is influenced by program supply and individual characteristics. Going back to institutional capacity,

note that, according to this framework, it is affected by program supply and technical inputs.

• We just looked at this framework by moving backward from the outcomes of interest through all the influencing factors. You could also read it

moving forward, however moving backwards allows you to keep the outcomes of interest in your mind when looking at relationships between

different factors. Also, note that that one factor can influence or be influenced by more than one other factor. In this example, you can also see that

program sustainability and institutional capacity influence each other.

• Additional Background

• The point at this stage is to show that this kind of framework explicitly incorporates factors external to the program in order to examine the

kind of impacts that contextual factors may have on outcomes of interest to the program.
 Conceptual Framework: Malaria Burden
External factors:
• Environmental (ecological, climate)
• Socio-economic (economic status,
Malaria
movement, occupation, housing condition, infection
war, population displacement, etc)
• Demographic ( age, immunity, gender)
Prevention:
• LLINs, IRS, IPT
Health care system: • Environmental
 Accessibility Malaria
management
 Affordability morbidity
 Quality of care Treatment:
 Efficiency Early diagnosis
 Demand/utilization
& treatment Malaria
mortality
Program factors: Malaria knowledge:
• Health policy • Cause
• Antimalarial drug policy • Prevention methods
• Support/partnership • Early treatment
• National MCP • Cultural beliefs
• Information
• Depending on the epidemiology of malaria, people are at risk of acquiring infection due to factors
related to environment, demographic and socio-economic status.

• Strong health care system (and malaria control program (MCP)) with adequate global and
national support, prevention and control of malaria using early diagnosis & prompt treatment, and
vector-control strategies have a significant impact on reducing malaria morbidity and mortality.

• Prevention methods inhibit the establishment of infection or suppress the progression of the
parasite after infection.

• Access to early diagnosis and prompt treatment with effective antimalarial drug significantly
reduce the severity of the illness, which will ultimately affect malaria mortality.

• A related factor is the perception of people about what causes malaria, their understanding of
early treatment with appropriate antimalarials, the use of personal preventive methods and
participation in disease prevention.
• Conceptual frameworks are typically shown as diagrams illustrating causal linkages
between the key components of a program and the outcomes of interest.

• For instance, in this example, the program, in addition to other donors, is supplying
health services, in order to increase service utilization, with the ultimate outcome of
improved health.

• By identifying the variables that factor into program performance and depicting the ways
that they interact, the results that can reasonably be expected from program activities are
outlined.

• Clarifying this process permits program designers to develop valid measures for
evaluating the success of the outcomes and also guides the identification of appropriate
indicators.
 2. Results Frameworks (RF)
 RF is a diagram that identifies steps or levels of results and illustrate the causal relationships
linking all levels of a program’s objectives. Results or strategic frameworks are a useful tool for
identifying and illustrating the focal points of a project for monitoring and evaluation: the results. A
results framework diagrams the relationships between the incremental results of the key
activities all the way up to the overall objective or goal of the intervention.

 Developing a framework focused on program results clarifies the points at which results can
be monitored and evaluated. In addition, results frameworks clearly depict the causal
relationships that the project design assumes will connect. For instance, the clinical training of
providers will connect with the provision of a higher quality of care at facilities where (more)
trained providers work, and thus ultimately lead to an improvement in health status or health
outcomes for the targeted population. The effectiveness of the project’s related activities can
be measured at each step along the way.
Purposes:
 Provides a clarified focus on the causal relationships that connect
incremental achievement of results to the comprehensive program
impact.

 Clarifies project/program mechanics and factors’ relationships that

suggest ways & means of objectively measuring the achievement of
desired ends.
Results Frameworks
Goal: Improved Health
Status

Strategic Objective: Improved Use of Health

Services or Practices

IR1: IR3:
IR2: Quality IR4: Demand
Access/Availability Sustainability

IR1.1:Commodities/ IR2.2: Provider

IR3.1: Policy IR4.1: Attitude
Facilities Performance

IR2.3: IR3.2: Health IR4.2:

IR1.2: Equity
Training/Supervision Care Finance Knowledge

IR2.4: IR4.3:
IR3.3: Private
Information Community
Sector
System Support
• This is a generic example of a typical results framework for a program
to improve use of health services or practices. The four themes of
access/availability, quality, sustainability and demand are usually
addressed.

• Notice that the goal and strategic objective appear at the top of the
framework, and under those are intermediate results or IRs. Under
each IR are subordinate intermediate results or sub-IRs that relate
directly to the intermediate results. For example, under IR2, factors
related to the information system, training and supervision of clinicians,
and provider performance lead to quality health services.
Results Framework: Malaria Control Program
SO1:
SO1:Reduced
Reduced
Malaria
MalariaBurden
Burden

IR1:
IR1:Improved
Improved IR2:
IR2:Improved
Improvedmalaria
malaria IR3:
IR3:Increased
Increasedaccess
access
malaria
malariaprevention
prevention epidemic
epidemicprevention
prevention to
toearly
earlydiagnosis
diagnosis&&
&&management
management prompt
prompttreatment
treatmentof
of
IR1.1 malaria
malaria
IR1.1Access
Accesstoto&& IR2.1
IR2.1Early
Earlydetection
detection
coverage
coverageby
byITNs
ITNs &&appropriate
appropriateresponse
response IR3.1
IR3.1Quality
Qualityof
of
increased
increased improved
improved care
careimproved
improved
IR1.2
IR1.2Improved
Improved IR2.2
IR2.2Epidemic
Epidemic IR3.2
IR3.2Efficiency
Efficiencyinin
access
accessto
toIPT
IPT preparedness
preparednessimproved
improved service
servicedelivery
delivery
IR1.3 improved
improved
IR1.3IRS
IRScoverage
coverage
increased
increasedinin IR2.3
IR2.3Surveillance
Surveillance
system
systemimproved
improved IR3.3
IR3.3Utilization
Utilizationof
of
epidemic
epidemicprone
proneareas
areas
care
careimproved
improved
IR1.4
IR1.4Use
Useof
ofsource
source
reduction/ IR2.4
IR2.4Early
Earlywarning
warning IR3.4
IR3.4Access
Accessto
to
reduction/larviciding
larviciding
increased system
systemstrengthened
strengthened services
servicesimproved
improved
increased
 The above framework shows the main critical areas for monitoring and evaluation of RBM control
interventions. The ultimate objective of RBM is to reduce the burden of malaria (i.e. mortality, morbidity
and economic losses). The reduction in burden will be achieved through control interventions undertaken
by the National Malaria Control Programs.

 The actual interventions, according to malaria epidemiology, will include the critical areas of: Prevention
(e.g. use of ITNs, indoor residual spraying, prevention of malaria during pregnancy); early diagnosis and
effective treatment of the disease, and prevention and control of epidemics in epidemic-prone
areas). Utilization of ITNs, application of indoor residual spraying (IRS), environmental management and
improved access to Intermittent Presumptive Treatment (IPT) will improve malaria prevention. Strong
surveillance system with appropriate early warning system to early detect and contain epidemics alleviate
mortality and morbidity associated with malaria epidemics. Quality of adequate care (technical and
perceived quality), efficiency in providing services, utilization of adequate care (prompt access and early
treatment seeking) and access to health care services are the critical areas that should be given attention
with regard to malaria treatment.
 3. Logical Frameworks (LF)
 LF is a diagram or matrix that illustrates the linear relationships between key program
inputs, activities, immediate results/outputs, and desired outcomes.

 Logical frameworks, also known as LogFRAMEs, are similar to results frameworks in that
they are program management tools that managed by results. USAID introduced the use
of logical frameworks to international development work but is now promoting the results
framework. These are still commonly used by projects funded by the United Nations
(UN). Other donor agencies including Australia AID, DFID, the UN and CIDA continue to
use logical frameworks.

 Logical frameworks logically summarize what the project intends to do and how, the key
assumptions, and outputs and outcomes that will be monitored and evaluated.
• A Logframe Matrix is a standardized table that summarizes the important
aspects of a project. They help set clear program objectives and define
indicators of success. They also outline the critical assumptions on which a
project is based, similar to the results framework. However, the logical
framework includes additional information, namely, it accounts for the
resources required to implement a program, the key activity clusters of a
program and the means for verifying project accomplishments.

• It is important to note that there is not just one way to create a logical
framework. Content and organization of the matrix generally vary by
organization.
Purposes of Logical frameworks
Summarizes what the project intends to do and how.

Summarizes key assumptions.

Summarizes outputs and outcomes that will be monitored and evaluated.

Note that:

• A Logframe Matrix is a standardized table that summarizes the important aspects of a project.

• Notice that the matrix contains all elements important to a program: goal, purpose, objectives,
outputs (meaning the measurable results produced by the program), and activities that will be
carried out to achieve the outputs. For each of these, indicators are specified and the means
for verifying these is also listed. In the last column, assumptions made for each program
element are also described.
 Logical Frameworks
Project Description Performance Indicators Means of Verification Assumptions

Goal: The broader development Measures of the extent to which a Sources of information and
impact to which the project sustainable contribution to the goal has been methods used to collect and
contributes - at a national and made. Used during evaluation. report it.
sectoral level.

Purpose: The development Conditions at the end of the project indicating Sources of information and Assumptions
outcome expected at the end of the that the Purpose has been achieved and that methods used to collect and concerning the
project. All components will benefits are sustainable. Used for project report it. purpose/goal linkage.
contribute to this completion and evaluation.

Component Objectives: The Measures of the extent to which component Sources of information and Assumptions
expected outcome of producing objectives have been achieved and lead to methods used to collect and concerning the
each component's outputs. sustainable benefits. Used during review and report it. component
evaluation. objective/purpose
linkage.

Outputs: The direct measurable Measures of the quantity and quality of Sources of information and Assumptions
results (goods and services) of the outputs and the timing of their delivery. Used methods used to collect and concerning the
project which are largely under during monitoring and review. report it. output/component
project management's control objective linkage.

Activities: The tasks carried out to Implementation/work program targets. Used Sources of information and Assumptions
implement the project and deliver during monitoring. methods used to collect and concerning the
the identified outputs. report it. activity/output linkage.
 Logical Frameworks: Malaria Control Program
Project Description Performance indicators Means of Assumptions
verification

Goal: To reduce malaria • Malaria prevalence • Annual reports • Political stability

morbidity and mortality by • All-cause under –five • Surveys • Environmental stability
50% by 2015 mortality rate • DSS (no natural disasters)
(INDEPTH)
• DHS
Overall Objective: To achieve % of individuals with access • Annual reports •Availability of effective
universal coverage of malaria to an LLIN in their HH • Surveys and affordable drugs
control interventions • Record reviews

Component Objective: % of individuals with access •Population- • Strong political support

•To achieve universal to an LLIN in their HH based survey •Availability of LLINs
coverage and increase use of % of individuals who slept •Health facility
LLINs under an LLIN the previous surveys
night • Community
surveys
• This example of a Logical Framework or LogFrame Matrix is for a malaria control
program. On this slide, notice there is one goal. Let’s follow the goal across the
matrix. The goal of reducing the burden of malaria morbidity and mortality and the
indicators include malaria prevalence and the All-cause under–five mortality rate

• This information will come from large scale population-based surveys such as the
DHS, MICS and MIS. There are a few assumptions relevant to the goal, including
environmental and political stability.

• You can also see the overall objective and one component objective on this slide.
 Logical Frameworks: Malaria Control Program
Project Description Performance Indicators Means of Assumptions
Verification

Outputs: Number of LLINs Activity Funds available

LLINs Distributed to target distributed to target reports/program for distribution and
population population records communication
campaign
LLIN hang up Number of individuals who
demonstrated to observed LLIN hang up Community
individuals in target demonstration support
population
Activities: Number of LLIN distribution Activity •Funds available
LLIN Distribution campaigns reports/program for distribution and
Campaigns records communication
Number of LLIN hang up campaign
LLIN hang up demonstrations
demonstrations
In this slide, two outputs from the malaria LogFrame Matrix has been selected for

illustration. One output is the LLINs distributed to the target population. Another

is that LLIN hag up is demonstrated. The indicators include Number of LLINs

distributed to target population and the Number of individuals who observed LLIN

hang up demonstration. Again, the data sources and the relevant assumptions

are also listed. Assumptions should be emphasized: They are something out of

your control, but that play an important role in the success of your plan.

Then, the two activities related to this output are listed. In this matrix, no inputs or

assumptions were listed, however in a complete matrix they would be included.

 How to develop a M&E framework
• Developing a monitoring and evaluation framework helps clarify which pieces of information to collect to evidence your story of
change. It is good practice to include people who will be collecting the data when you develop your framework. You could also
involve beneficiaries, volunteers, trustees, partner organisations or funders. Ideally, write your framework before your project starts
so you can make sure you are collecting appropriate data from the beginning.

1. Use a planning tool. ...

2. Describe your outputs. ...

3. Set output indicators. ...

4. Describe your outcomes. ...

5. Set outcome indicators. ...

6. Plan how to measure soft outcomes. ...

7. Test knowledge and awareness carefully. ...

8. Review and prioritize your outcome indicators.

9. Consider whether you need to evidence impact

10.Plan your process monitoring

 Part two: M&E Systems
• A project M&E system covers all the work carried out during or after a project to define, select,
collect, analyze and use information. It is where everything comes together, from the initial selection
of objectives and indicators through to the final evaluation of a project. M&E systems in
straightforward, timebound projects typically contain several common elements.

• A project M&E system requires a context. This means defining key features of the system such as
its scope and purpose, or key principles which will govern its operation.

• A well-functioning M&E system helps guide the intervention strategy and ensure effective
operations for all key stakeholders. It is one part of overall project management. The M&E system
will itself need to be monitored and updated regularly during the life of the project.

• A key question for any project M&E system is ‘what do we need to know in order to monitor and
evaluate the project?’
 Inner cycle is about Monitoring, which involves 6 steps

1. Identify Project Objectives and Strategies: This part could be referred to project

design/planning stage. If it is not clearly developed during planning or design stage, this time

M&E officer and management staff need to determine it.

2. Setting Indicators: This part is also referred to the project design/planning stage that

organization or institution had developed especially indicators and targets or milestones that

the project or program wants to be achieved.

3. Selecting the Tools and Techniques: There are several tools and techniques which are

used for data collection including Participatory Rural Appraisal (PRA) or Rapid Rural Appraisal

(RRA), Agri-Ecosystem Analysis (AEA) tools etc. Those tools could be used either a

participatory or empowerment way depends on thematic issues, context of the project or program.
03/05/2024 36
 Inner cycle is about Monitoring, which involves 6 steps (Cont)

4. Collecting Data: data collection which is referred to secondary and primary

data collection using tools and techniques above. Team of data collectors,
data enumerators, data entry and interpreters must be mobilized and worked.

5. Analyze the Results: Once the data are collected or collated, then data
need to be analysed and written in a narrative way and easy to understand by
all stakeholders.

6. Use the Learning to Improve: After all findings are presented key lessons
needed to be taken and learned to improve next phase of project or program
design/planning again.
03/05/2024 37
Outer cycle is about Evaluation, which involves 5 key criteria for any project or program evaluation

1. Relevance: The extent to which the objectives of a project are consistent with the target group’s priorities & the

recipient & donors' policies. The assessment will look at the project’s services is met/responded to beneficiaries needs.

2. Effectiveness: The assessment will look at whether the project or program has achieved its stated objectives and goal

during that setting timeframe (or life cycle of the project).

3. Efficiency: The assessment will look at how the organization or project staff had utilized resources to achieve such

objectives and goal. Some time misused of funds or materials resources were not make the project or program

successful.

4. Sustainability: The assessment will look at the perception or views of beneficiaries and other stakeholders related to

whether or not the project or program can be sustainable after the project completion or after the project phased out

from that area.

5. Impact: The assessment will look at the positive changes arising from the project or program including change in

behaviors, practices, change in living condition and change in policies.

03/05/2024 38
 Elements of a Good M&E System

1. A capable Monitoring and Evaluation staff unit

2. Clear goals and objectives of the program or project

3. A core set of indicators and targets

4. A plan for data collection and analysis

5. A plan for data dissemination

39
 Five Elements of a Good M&E System (cont)
M&E Unit
Not So Good Good
 No functioning unit for M&E in the organization/  Established M&E unit within the Department, Ministry
Ministry/ department and organization
 1 or 2 persons responsible for the whole country  Specific expertise in or affiliated with the unit: (M&E,
 Very limited resources for M&E behavioral, statistics, data dissemination)
 No formalized links with technical and other  Budget (10% of the national budget)
resources  Formalized links with the research institutions, leading
NGOs and donors

Clear Goal and Objective of program/project

 National strategic plan has no specific goals  Well-defined national program goals and targets -
and objectives M&E plan
 No system of ongoing assessment with  Regular reviews/evaluations of the progress of the
programs reviews and built-in evaluation implementation of the national program plans
 Limited coordination with districts and  Guidelines and guidance to districts and or
provinces provinces for M&E
 Limited coordination between sectors  Guidelines for linking M&E to multiple sectors
 Donor-driven M&E system  Co-ordination of national and donor M&E needs

40
 Five Elements of a Good M&E System (cont)
A Set of Indicators and Targets
 No indicators or indicators that cannot be measured  A set of priority indicators and additional indicators that
 Indicators that cannot be compared with past indicators or cover program monitoring, program outcomes and impact
with other countries - M&E plan
 Indicators are only used for donors and each donor has its  Selection of indicators through process of involving
own set of indicators multiple stakeholders and maintaining relevance and
 Indicators are irrelevant to those who collect the data comparability
 Each sector uses its own indicator  Utilization of past and existing data collection efforts to
assess national trends.
A Plan for data collection and analysis
 M&E is an ad-hoc activity without a plan, mostly driven by  An overall national level data collection and analysis plan,
donors linked to the national strategic plan
 Data are collected but not analyzed sufficiently / utilized  A plan to collect data and analyze indicators at different
 There is no systematic monitoring of program inputs and levels of M&E (program monitoring)
outputs.  Second generation surveillance, where behavioral data
are linked to surveillance data.
A Plan for data dissemination
 Dissemination is ad hoc and not planned or coordinated  Overall national level data dissemination plan
 Annual surveillance report is much delayed not user friendly  Well-disseminated informative annual report of the M&E
and not well disseminated unit
 Dissemination to the districts and regions is not done  Annual meetings to disseminate and discuss M&E and
 Dissemination activities are donor driven research findings with policy-makers and planners
 Clearinghouse / Resource centre at national level
41
 Key steps in setting up M&E system

1. Identify Project Objectives and Strategies

2. Setting Indicators

3. Selecting the Tools and Techniques

4. Collecting Data

5. Analyse the Results

6. Use the Learning to Improve

42
 Key Steps in setting up M&E system (Cont)

Why do we need M&E and how comprehensive should

Step 1: Identify/
our M&E system be? This should also be linking to
establish purpose program or project strategy that included Goals,
and scope Objectives, Activities/Process, Inputs, Outcomes, and
Impacts.

What do we need to know to monitor and

Step 2:
Identify performance evaluate the project in order to manage it well?
question, information
and what performance progresses be
needs and indicators
monitored?

43
 Key Steps in setting up M&E system (Cont)

How will the required information be gathered and

Step 3: Planning organized? M&E data collection in different methods and
information gathering and tools should also be considered.
organization

Step 4: Planning critical How will we make sense of the information gathered
reflection process and and use it to make improvements for our project or
events program?

44
 Key Steps in setting up M&E system (Cont)

How and to whom do we want to communicate what in terms of our

Step 5: Planning for quality project activities and processes?
communication and
reporting

Step 6: Planning for the What is needed to ensure our M&E system actually works? Use the
necessary conditions and learning to improve our project is the best way in this step.
capacities

45
 2.4 Understanding Indicators

• Once the framework is finalized, the next step in completing the monitoring
and evaluation framework is selecting indicators.

• Indicators are signs of progress – they are used to determine whether the
programme/intervention is on its way to achieving its objectives and goal.

• An indicator is a specific, observable and measurable characteristic that

can be used to show changes or progress a programme is making toward
achieving a specific outcome.

• Indicators provide M&E information crucial for decision making at every

level and stage of program implementation.
 What are indicators?
 There should be at least one indicator for each outcome. The indicator should be
focused, clear and specific. The change measured by the indicator should
represent progress that the programme hopes to make.

 An indicator should be defined in precise, unambiguous terms that describe

clearly and exactly what is being measured. Where practical, the indicator should
give a relatively good idea of the data required and the population among whom
the indicator is measured.

 Indicators do not specify a particular level of achievement -- the words

“improved”, “increased”, or “decreased” do not belong in an indicator.
• An indicator is a variable that measures one aspect of a program or project that is directly

related to the program’s objectives. Let’s take a moment to go over each piece of this definition.

• First, an indicator is a variable whose value changes from the baseline level at the time the

program began to a new value after the program and its activities have made their impact felt. At

that point, the variable, or indicator, is calculated again.

• Second, an indicator is a measurement. It measures the value of the change in meaningful units

that can be compared to past and future units. This is usually expressed as a percentage or a

number.

• Third, an indicator focuses on a single aspect of a program or project. This aspect may be an

input, an output, or an overarching objective, but it should be narrowly defined in a way that

captures this one aspect as precisely as possible.

• A reasonable guideline recommends one or two indicators per result, at least one
indicator for each activity, but no more than 10 to 15 indicators per area of
significant program focus.

• Indicators can be either quantitative or qualitative.

• Quantitative indicators are numeric and are presented as numbers or percentages.

• Qualitative indicators are descriptive observations and can be used to supplement

the numbers and percentages provided by quantitative indicators. They complement
quantitative indicators, by adding a richness of information about the context in
which the program has been operating. Examples are “availability of a clear,
strategic organizational mission statement” and “existence of a multiyear
procurement plan for each product offered.”
• Indicators of program inputs measure the specific resources that go into carrying out a

project or program (for example, amount of funds allocated to the health sector

annually).

• Indicators of outputs measure the immediate results obtained by the program (for

example, number of multivitamins distributed or number of staff trained).

• Indicators of outcomes measure whether the outcome changed in the desired

direction and whether this change signifies program “success” (for example,

contraceptive prevalence rate or percentage of children 12 to 23 months who received

DTP3 immunization by 12 months of age).

 Characteristics of good indicators
Valid: accurate measure of a behavior, practice, task that is the expected output or
outcome of the intervention

Reliable: consistently measurable over time, in the same way by different observers

Precise: operationally defined in clear terms

Measurable: quantifiable using available tools and methods

Timely: provides a measurement at time intervals relevant and appropriate in terms

of programme goals and activities

Programmatically important: linked to the programme or to achieving the

programme objectives (Gage and Dunn, 2009)
 Process versus result/impact indicators
•It is important to remember the difference between process and results indicators.

• Process Indicators are used to monitor the number and types of activities carried out. Examples
include:

 The number and types of services provided

 The number of people trained

 The number and type of materials produced and disseminated

 The number and percentage of female clients screened

• Results Indicators are used to evaluate whether or not the activity achieved the intended objectives
or results. Examples include:

 Selected indicators of knowledge, attitudes and practices as measured by a survey

 The perceptions of survivors about the quality and benefits of services provided by an organization or
institution as measured by individual interviews (Bott, Guedes and Claramunt, 2004).
 2.4. Setting Targets
• Once a set of goals and objectives has been established in accordance with the agreed vision, it will be

important to set Targets. Indicators are used to measure achievement. In other words, they measure the

actual results from an activity, project or programme. Targets are the quantitative goal or objective for an

activity, project or programme. In other words, they are the expected results. Essentially, indicators are

used to determine if targets are being met.

• Targets help define, in specific and measurable terms, the desired outcomes of the strategy. They should

be incorporated into a vision statement to contribute to a "SMART" objective. A vision or a vision

statement can have multiple goals and each goal can have many objectives. At the same time, each

objective can be linked to multiple targets and each target could be linked to various KPIs.

• Developing targets and indicators needs to be undertaken concurrently with the formulation of objectives.

Where targets and indicators are developed in isolation from the objectives, there is a risk that the

indicators start to drive rather than support the decision-making process.

• The benefits of target-setting
• If you've identified the key areas that drive your business performance and found a way to measure them,

then a natural next step is to start setting performance targets to give everyone in your business a clear

sense of what they should be aiming for.

• Strategic visions can be difficult to communicate, but by breaking your top-level objectives down into

smaller concrete targets you'll make it easier to manage the process of delivering them. In this way, targets

form a crucial link between strategy and day-to-day operations.

• Targets help to set up a clear course of action and guide future direction. By providing relevant

benchmarks, targets provide a measure of how successful the strategy may have been and determine

required adjustment to improve performance and results. As M&E is a multi-faceted area of activity,

targets should be clear and there should be no ambiguity about the objective that is being prioritized.
• This ensures that stakeholders understand how the different objectives are being balanced.

• This in turn, can help secure stakeholders’ buy-in and support.

• These targets are monitored using KPIs, which are inherently linked to the
objectives and goals.

• KPIs are indicators used to assess whether stakeholders are on target as they
work towards achieving the vision’s goals and objectives. Indicators may need to
be refined over time and improved to better reflect stakeholders’ needs.

• For example, the policy priorities may change due to new and emerging critical
challenges or there could be new developments in statistical methodologies and
data availability, creating new opportunities for indicator refinement.

• Thus, some level of flexibility should be allowed when developing indicators.

• This aspect is elaborated further under the Monitoring and Evaluation step.
• When selecting the KPIs to be tracked, stakeholders’ involvement is necessary. Targets
and KPIs must also be accepted by all the parties that are expected to take responsibility for
their achievement. Consulting stakeholders, including stakeholders identified during the
Diagnosis and BLS, is required to generate support for the KPIs and targets and to discuss
the measures that are most important.

• Stakeholders need to decide who should be compiling information about the relevant
measures and agree on how to link these measures to the strategies developed and
implemented to achieve sustainable freight transportation. Consultations could also help
address potential data gaps and double counting.

• Efforts should aim to improve the M&E-related data sources, availability, granularity, reliability
and quality. Improved data systems can support the development of useful indicators, which
in turn, are crucial enablers of sound and informed decision-making processes.
 How to set targes?
• While there is no standard formula for setting targets, there are several factors to consider when
determining them. The followings are some of these considerations:

 Baseline data: What is the situation at the outset of the activity, project or programme?

 Historical trends: What pattern of change has occurred over time? Is the same pattern likely to
continue?

 Stakeholders’ expectations: What do key stakeholders (e.g. government officials and

programme implementers) believe can/should/must be accomplished?

 Expert opinions and research findings: What do the experts think about targets? What has
previous research indicated that is relevant to target setting?

 Performance of similar activities, projects or programmes: How have similar efforts

performed at other times and/or in other settings? With other implementing agencies and/or
partners?
End of Chapter 2