MONITORING & EVALUATION

Chapter 1: OVERVIEW OF MONITORING AND

EVALUATION

(i) What is Monitoring and Evaluation

 Monitoring and Evaluation is a process of continued gathering of information and its
analysis, in order to determine whether progress is being made towards pre-specified goals and
objectives, and highlight whether there are any unintended (positive or negative) effects from a
project/programme and its activities.

(ii) What is a Monitoring?

 Monitoring is a continuous process of collecting, analyzing, documenting, and reporting
information on progress to achieve set project objectives. It helps identify trends and patterns,
adapt strategies and inform decisions for project or programme management.

(iii) What is Evaluation?

 Evaluation is a periodic assessment, as systematic and objective as possible, of an on- going
or completed project, programme or policy, its design, implementation and results. It
involves gathering, analysing, interpreting and reporting information based on credible data.
The aim is to determine the relevance and fulfilment of objectives, developmental
efficiency, effectiveness, impact and sustainability.

(iv) Purpose/Importance of Monitoring and Evaluation

Timely and reliable M&E provides information to:
 Support project/programme implementation with accurate, evidence-based reporting that
informs management and decision-making to guide and improve project/programme
performance.
 Contribute to organizational learning and knowledge sharing by reflecting upon and
sharing experiences and lessons.
 Uphold accountability and compliance by demonstrating whether or not our work has been
carried out as agreed and in compliance with established standards and with any other
stakeholder requirements
 Provide opportunities for stakeholder feedback,.
 Promote and celebrate project/program work by highlighting accomplishments and
achievements, building morale and contributing to resource mobilization.
 Strategic management in provision of information to inform setting and adjustment of
objectives and strategies.
 Build the capacity, self-reliance and confidence stakeholders, especially beneficiaries and
implementing staff and partners to effectively initiate and implement development
initiatives.

v) Characteristics of monitoring and evaluation

Monitoring tracks changes in program performance or key outcomes over time. It has the following
characteristics:
 Conducted continuously
 Keeps track and maintains oversight
 Documents and analyzes progress against planned program activities
 Focuses on program inputs, activities and outputs
 Looks at processes of program implementation
 Considers program results at output level
 Considers continued relevance of program activities to resolving the health problem
 Reports on program activities that have been implemented
 Reports on immediate results that have been achieved

Page 1
Evaluation is a systematic approach to attribute changes in specific outcomes to program
activities. It has the following characteristics:
 Conducted at important program milestones

 Provides in-depth analysis

 Compares planned with actual achievements
 Looks at processes used to achieve results
 Considers results at outcome level and in relation to cost
 Considers overall relevance of program activities for resolving health problems
 References implemented activities
 Reports on how and why results were achieved
 Contributes to building theories and models for change
 Attributes program inputs and outputs to observed changes in program outcomes and/or
impact

Page 2
(v) Key benefits of Monitoring and Evaluation
a. Provide regular feedback on project performance and show any need for ‘mid-
course’ corrections
b. Identify problems early and propose solutions
c. Monitor access to project services and outcomes by the target population;
d. Evaluate achievement of project objectives, enabling the tracking of progress
towards achievement of the desired goals
e. Incorporate stakeholder views and promote participation, ownership and
accountability
f. Improve project and programme design through feedback provided from baseline, mid-
term, terminal and ex-post evaluations
g. Inform and influence organizations through analysis of the outcomes and impact of
interventions, and the strengths and weaknesses of their implementation, enabling
development of a knowledge base of the types of interventions that are successful (i.e.
what works, what does not and why.
h. Provide the evidence basis for building consensus between stakeholders

Page 3
 Chapter 2 SELECTING INDICATORS, BASELINES AND TARGETS
a) The indicator: “An indicator is defined as a quantitative measurement of an objective to be
achieved, a resource mobilised, an output accomplished, an effect obtained or a context
variable (economic, social or environmental)”. precise information needed to assess whether
intended changes have occurred. Indicators can be either quantitative (numeric) or qualitative
(descriptive observations). Indicators are typically taken directly from the logframe, but should
be checked in the process to ensure they are SMART (specific, measurable, achievable,
relevant and time-bound).
b) The Indicator definition- key terms in the indicator that need further detail for precise and
reliable measurement.
c) The methods/sources- identifies sources of information and data collection methods and tools,
such as the use of secondary data, regular monitoring or periodic evaluation, baseline or
endline surveys, and interviews.
d) The frequency/schedules -how often the data for each indicator will be collected, such as
weekly, monthly, quarterly, annually, etc.
e) The person(s) responsible- lists the people responsible and accountable for the data
collection and analysis, e.g. community volunteers, field staff, project/programme
managers, local partner(s) and external consultants.
f) The information use/audience - identifies the primary use of the information and its
intended audience. Some examples of information use for indicators include:
• Monitoring project/programme implementation for decision-making
• Evaluating impact to justify intervention
• Identifying lessons for organizational learning and knowledge-sharing
• Assessing compliance with donor or legal requirements
• Reporting to senior management, policy-makers or donors for strategic planning
• Accountability to beneficiaries, donors and partners
• Advocacy and resource mobilization.
g) Types of Indicators

 Context indicators which measure an economic, social or environmental variable

concerning an entire region, sector, or group and the Project location, as well as relevant
national and regional policies and programs.. The situation before the project starts, the
(baseline) data, primarily from official statistics.
 Input indicators include indicators that measure the human and financial resources,
physical facilities, equipment and supplies that enable implementation of a program
 Process indicators reflect whether a program is being carried out as planned and how well
program activities are being carried out.
 Output indicators which relate to activities, measured in physical or monetary units
/results of program efforts (inputs and processes/activities) at the program level.
 Outcome indicators measure the program’s level of success in improving service
accessibility, utilization or quality.
 Result indicators- direct and immediate effect arising from the project activities that
provide information on changes of the direct project beneficiaries.
 Impact indicators refer to the the long-term, cumulative effects of programs over time,
beyond the immediate and direct effects on beneficiaries
 Exogenous indicators are those that cover factors outside the control of the project but
which might affect its outcome.
 Proxy indicators – an indirect way to measure the subject of interest

Page 4
h) Characteristics of Good Indicators.
a) Specific – focused and clear
b) Measurable - quantifiable and reflecting change
c) Attainable - reasonable in scope and achievable within set time-frame
d) Relevant - pertinent to the review of performance
e) Time-Bound/Trackable - progress can be charted chronologically Also
be CREAM: Clear, Relevant, Economical, Adequate and Monitor-able.

i) Baselines and Targets

 A baseline is qualitative or quantitative information that provides data at the
beginning of, or just prior to, the implementation of an intervention.
 Targets are established for each indicator by starting from the baseline level, and by
including the desired level of improvement in that indicator

Page 5
Chapter 3: FRAMEWORKS FOR EVALUATION - THE LOGICAL FRAMEWORK
APPROACH (LFA)
Four types of frameworks dominate the M&E field:
a) Conceptual frameworks are also known as theoretical or causal frameworks.
b) Results-based frameworks are also known as strategic frameworks and serve as a
management tool with an emphasis on results. The purpose of results frameworks is to
increase focus, select strategies, and allocate resources accordingly.
Impact The higher-order objective to which a development intervention is intended to contribute.
Outcome The likely or achieved short-term and medium-term effects of an intervention’s outputs.
Output The products, capital goods and services which result from a development intervention; may also
include changes resulting from the intervention which are relevant to the achievement of outcomes.
Activity Actions taken or work performed through which inputs, such as funds, technical assistance and other
types of resources are mobilized to produce specific outputs.
Inputs The financial, human, and material resources used for the development intervention.

c) Logical frameworks are also known as LogFRAMEs and are commonly used to help set
clear program objectives and define indicators of success. They also outline the critical
assumptions on which a project is based, similar to the results framework.
d) Logic models are also known as M&E frameworks are commonly used to present a clear
plan for the use of resources to meet the desired goals and objectives. They are a useful tool
for presenting programmatic and evaluation components.
The choice of a particular type of framework—whether a conceptual framework, results
framework, logical framework or logic model—depends on the program’s specific needs, the M&E
team’s preferences and donor requirements.
In particular, the LFA is a systematic planning procedure for complete project cycle management, a
participatory Planning, Monitoring & Evaluation tool;

 A tool for planning a logical set of interventions;

 A tool for appraising a Programme document;
 A concise summary of the Programme;
 A tool for monitoring progress made with regard to delivery of outputs and activities;
 A tool for evaluating impact of Programme outputs, e.g. progress in achieving
purpose and goal.

Page 6
Narrative summary Objectively verifiable Means of verification Assumptions/Risks- what
A snapshot of the different levels of indicators (OVI) - how (MOV) - how will assumptions underlie the
the project objectives— known as the will we know we've been we check our structure of our project and
“hierarchy of objectives”. successful? reported results? what is the risk they will not
prevail?
Goal (Impact)- Longer-term
effects/General or overall objective
Purpose- - why are we doing this?
direct and immediate
effects/objectives/Outcomes/Results
Outputs - what are the
deliverables? – goods and
services produced/operational
objectives
Activities- what tasks will we Inputs Cost
undertake to deliver the outputs? By what means do we What does it cost
carry out the activities

Pre-conditions
What needs to be fulfilled
before activities can start

Page 7
 Chapter 4: MONITORING CRITERIA
a) Project monitoring & control cycle.
To achieve effective control over project implementation, it is necessary to assess the progress from
time at regular intervals in terms of physical completion of scheduled activities, actual cost incurred
in performing those activities ad achievement of desired performance levels by comparing the
status with the plans to find deviations. This assessment process is known as ‘monitoring’.

PLAN COMPARE ACTUAL

STATUS

VARIANCES

NO YES

REVISED SCHEDULES,BUDGETS ESTIMATES TO COMPLETE

ACTION PLAN

Key elements of project monitoring and control

 Project Status reporting
 Conducting a project review with stakeholders
 Controlling schedule variances
 Controlling scope and change requests
 Controlling budget
 Tracking and mitigating risks

Page 8
 b) Types of monitoring
A project/programme usually monitors a variety of things according to its specific informational needs.
These monitoring types often occur simultaneously as part of an overall monitoring system commonly
found in a project/programme monitoring system.
TAB 1: Common types of monitoring
 Results monitoring: Tracks effects and impacts to determine if the project/programme is on
target towards its intended results (inputs, activity, outputs, outcomes, impact,
assumptions/risks monitoring) and whether there may be any unintended impact (positive or
negative
 Process (activity) monitoring : Tracks the use of inputs and resources, the progress of activities,
how activities are delivered – the efficiency in time and resources and the delivery of
outputs
 Compliance monitoring: Ensures compliance with, say, donor regulations and expected results,
grant and contract requirements, local governmental regulations and laws, and ethical
standards.
 Context (situation) monitoring: Tracks the setting in which the project/programme operates,
especially as it affects identified risks and assumptions, and any unexpected considerations that
may arise, including the larger political, institutional, funding, and policy context that
affect the project/programme.
 Beneficiary monitoring: Tracks beneficiary perceptions of a project/programme. It includes
beneficiary satisfaction or complaints with the project/programme, including their participation,
treatment, access to resources and their overall experience of change.
 Financial monitoring: Accounts for costs by input and activity within predefined categories of
expenditure, to ensure implementation is according to the budget and time frame.
 Organizational monitoring: Tracks the sustainability, institutional development and capacity
building in the project/programme and with its partners.

c) Monitoring Questions and the LogFrame

Page 9
EVALUATION CRITERIA FOR PROJECTS
a) Five Part Evaluation Criteria
 Relevance - Was/is the project a good idea given the situation to improve? Was the
logic of the project correct? Why or Why Not? -The validity of the Overall Goal and
Project Purpose at the evaluation stage.
 Effectiveness - Have the planned results been achieved? Why or Why Not? -The degree to
which the Project Purpose has been achieved by the project Outputs.
 Efficiency - Have resources been used in the best possible way? Why or Why Not? -The
productivity in project implementation. The degree to which Inputs have been
converted into Outputs.
 Impact - To what extent has the project contributed towards its longer term goals? Why or
Why Not? Have there been any unanticipated positive or negative consequences of the
project? Why did they arise? -Positive and negative changes produced, directly or
indirectly, as a result of the Implementation of the project.
 Sustainability – Can the outcomes be sustained after the project funding to ensure
continued impacts? Why or Why Not? -The durability of the benefits and development
effects produced by the project after its completion.

b) Evaluation Questions and the LogFrame

Page 10
 Chapter 5. TYPES OF EVALUATION
Three ways of classifying:
 When it is done - Ex-ante evaluation; Formative evaluation; Summative – end of
project, and Ex-Post evaluation.
 Who is doing it - External evaluation; Internal evaluation or self-assessment
 What methodology or technicality is used- Real-time evaluations (RTEs); Meta-
evaluations; Thematic evaluations; Cluster/sector evaluations; Impact evaluations
The details are as follows: -
a) Ex–ante evaluation: Conducted before the implementation of a project as part of the
planning. Needs assessment determines who needs the program, how great the need is, and
what might work to meet the need. Implementation(feasibility)evaluation monitors the
fidelity of the program or technology delivery, and whether or not the program is
realistically feasible within the programmatic constraints
b) Formative evaluation: Conducted during the implementation of the project. Used to
determine the efficiency and effectiveness of the implementation process, to improve
performance and assess compliance. Provides information to improve processes and learn
lessons. Process evaluation investigates the process of delivering the program or
technology, including alternative delivery procedures. Outcome evaluations
investigate whether the program or technology caused demonstrable effects on specifically
defined target outcomes. Cost-effectiveness and cost-benefit analysis address questions of
efficiency by standardizing outcomes in terms of their dollar costs and values
c) Midterm evaluations are formative in purpose and occur midway through
implementation.
d) Summative evaluation: Conducted at the end of the project to assess state of project
implementation and achievements at the end of the project. Collate lessons on content and
implementation process. Occur at the end of project/programme implementation to assess
effectiveness and impact.
e) Ex-post evaluation: Conducted after the project is completed. Used to assess
sustainability of project effects, impacts. Identifies factors of success to inform other
projects. Conducted sometime after implementation to assess long-term impact and
sustainability.
f) External evaluation: Initiated and controlled by the donor as part of contractual
agreement. Conducted by independent people – who are not involved in implementation.
Often guided by project staff
g) Internal or self-assessment: Internally guided reflective processes. Initiated and
controlled by the group for its own learning and improvement. Sometimes done by
consultants who are outsiders to the project. Need to clarify ownership of information before
the review starts
h) Real-time evaluations (RTEs): are undertaken during project/programme
implementation to provide immediate feedback for modifications to improve on-going
implementation.
i) Meta-evaluations: are used to assess the evaluation process itself. Some key uses of meta-
evaluations include: take inventory of evaluations to inform the selection of future
evaluations; combine evaluation results; check compliance with evaluation policy and good
practices; assess how well evaluations are disseminated and utilized for organizational
learning and change, etc.
j) Thematic evaluations: focus on one theme, such as gender or environment, typically across
a number of projects, programmes or the whole organization.
k) Cluster/sector evaluations: focus on a set of related activities, projects or programmes,
typically across sites and implemented by multiple organizations
l) Impact evaluations: is broader and assesses the overall or net effects -- intended or
unintended -- of the program or technology as a whole focus on the effect of a
project/programme, rather than on its management and delivery. Therefore, they typically
occur after project/programme completion during a final evaluation or an

Page 11
 ex-post evaluation. However, impact may be measured during project/programme
implementation during longer projects/programmes and when feasible.

EVALUATION MODELS AND APPROACHES

 Behavioral Objectives Approach. -“Is the program, product, or process achieving its
objectives?” Focuses on the degree to which the objectives of a program, product, or
process have been achieved.
 The Four-Level Model or Kirkpatrick Model -“What impact did the training have on
participants in terms of their reactions, learning, behavior, and organizational results?” Often
used to evaluate training and development programs and focuses on four levels of training
outcomes: reactions, learning, behavior, and results.
o Reaction - Measures trainees’ valuable experience, feel good about the instructor, the
topic, the material, its presentation, and the venue.
o Learning- How much has their knowledge increased as a result of the training?
o Behavior-trainees have changed their behavior, based on the training they received.
o Results- good for business, good for the employees, or good for the bottom line.
 Management Models-“What management decisions are required concerning the
program”. The evaluator’s job is to provide information to management to help them in
making decisions about programs, products, etc. Daniel Stufflebeam’s CIPP Model has been
very popular. CIPP stands for context evaluation, input evaluation, process evaluation, and
product evaluation. Context evaluation includes examining and describing the context of
the program you are evaluating, conducting a needs and goals assessment, determining the
objectives of the program, and determining whether the proposed objectives will be
sufficiently responsive to the identified needs. It helps in making program planning
decisions. Input evaluation includes activities such as a description of the program inputs
and resources, a comparison of how the program might perform compared to other
programs, a prospective benefit/cost assessment (i.e., decide whether you think the benefits
will outweigh the costs of the program, before the program is actually implemented), an
evaluation of the proposed design of the program, and an examination of what alternative
strategies and procedures for the program should be considered and recommended. Process
evaluation includes examining how a program is being implemented, monitoring how the
program is performing, auditing the program to make sure it is following required legal and
ethical guidelines, and identifying defects in the procedural design or in the implementation
of the program. Product evaluation includes determining and examining the general and
specific anticipated and unanticipated outcomes of the program (i.e., which requires using
impact or outcome assessment techniques).
 Responsive Evaluation. -“What does the program look like to different people?” - Calls for
evaluators to be responsive to the information needs of various audiences or stakeholders.
 Goal-Free Evaluation. -“What are all the effects of the program, including any side
effects?”-Fcuses on the actual outcomes rather than the intended outcomes of a program. Thus,
the evaluator is unaware of the program’s stated goals and objectives.
 Adversary/Judicial Approaches. “What are the arguments for and against the program?”-
These Adopts the legal paradigm to program evaluation, where two teams of evaluators
representing two views of the program’s effects argue their cases based on the evidence (data)
collected. Then, a judge or a panel of judges decides which side has made a better case and
makes a ruling.
 Consumer-Oriented Approaches. “Would an educated consumer choose this program or
product?”- helps consumers to choose among competing programs or products.
 Expertise/Accreditation Approaches. “How would professionals rate this program?”- The
accreditation model relies on expert opinion to determine the quality of programs. The purpose
is to provide professional judgments of quality.
 Utilization-Focused Evaluation. “What are the information needs of stakeholders, and how
will they use the findings?”- Evaluation done for and with specific, intended primary users for
specific, intended uses. ” Assumes stakeholders will have a high degree of involvement in
many, if not all, phases of the evaluation. The major question being addressed is,

Page 12
 Participatory/Collaborative Evaluation.- “What are the information needs of those closest
to the program?”- Engaging stakeholders in the evaluation process, so they may better
understand evaluation and the program being evaluated and ultimately use the evaluation
findings for decision-making purposes.
 Empowerment Evaluation. “What are the information needs to foster improvement and self-
determination?”. Use of evaluation concepts, techniques, and findings to foster
improvement and self-determination, a catalyst for learning in the workplace a social
activity in which evaluation issues are constructed by and acted on by organization
members
 Organizational Learning. “What are the information and learning needs of individuals,
teams, and the organization in general?” ongoing and integrated into all work practices
 Theory-Driven Evaluation.- “How is the program supposed to work? What are the
assumptions underlying the program’s development and implementation?”- Focuses on
theoretical rather than methodological issues to use the “program’s rationale or theory as the
basis of an evaluation to understand the program’s development and impact” using a plausible
model of how the program is supposed to work.
 Success Case Method. “What is really happening?”- focuses on the practicalities of
defining successful outcomes and success cases and uses some of the processes from theory-
driven evaluation to determine the linkages, which may take the form of a logic model, an
impact model, or a results map. Evaluators using this approach gather stories within the
organization to determine what is happening and what is being achieved. The major question
this approach asks is,

Page 13
 Chapter 6: THE EVALUATION PROCESS
Evaluation operates within multiple domains and serves a variety of functions at the same time.
Moreover it is subject to budget, time and data constraints that may force the evaluator to sacrifice
many of the basic principles of impact evaluation design. Before entering into the details of
evaluation methods it is important for the reader to have a clear picture of the way an evaluation
procedure works.

(i) The M&E Plan/strategy

A comprehensive planning document for all monitoring and evaluation activities within a
program. This plan documents the key M&E questions to be addressed: what indicators will
be collected, how, how often, from where, and why; baseline values, targets, and
assumptions; how data are going to be analyzed/interpreted; and how/how often report will be
developed and distributed.

Typically, the components of an M&E plan are:

 Establishing goals and objectives
 Setting the specific M&E questions
 Determining the activities to be implemented
 The methods and designs to be used for monitoring and evaluation
 The data to be collected
 The specific tools for data collection
 The required resources
 The responsible parties to implement specific components of the plan
 The expected results
 The proposed timeline

(ii) Monitoring And Evaluation Cycle

Step 1 – Identify the purpose and scope of the M&E system

 Formulating objectives
 Selecting Indicators
 Setting baselines and targets
Step 2 – Plan for data collection and management

Page 14
  Major sources of data- secondary data, primary data-sample surveys, project output data,
qualitative studies- PRA, mapping, KIIs, FGDs, observation, checklists, external
assessments, participatory assessments
 planning for data collection - prepare data collection guidelines, pre-test data
collection tools, train data collectors, address ethical issues

Step 3 – Plan for data analysis

Step 4 – Plan for information reporting and utilization
Step 5 – Plan for M&E human resources and capacity building Step 6
– Prepare the M&E budget

(iii) Setting up an M&E system often involves the following aspects

a) Assess the existing readiness and capacity for monitoring and evaluation
b) Review current capacity within (or outsourced without) the organization and its partners
which will be responsible for project implementation, covering: technical skills,
managerial skills, existence and quality of data systems, available technology and
existing budgetary provision.
c) Establish the purpose and scope
Why is M&E needed and how comprehensive should the system be?
What should be the scope, rigour and should the M&E process be participatory?
d) Identify and agree with main stakeholders the outcomes and development
objective(s).
Set a development goal and the project purpose or expected outcomes, outputs,
activities and inputs. Indicators, baselines and targets are similarly derived
e) Select key indicators i.e the qualitative or quantitative variables that measure project
performance and achievements for all levels of project logic with respect to inputs,
activities, outputs, outcomes and impact, as well as the wider environment,
requiring pragmatic judgment in the careful selection of indicators.
f) Developing and Evaluation Frame work - set out the methods, approaches and
evaluation designs ( Experimental, Quasi-Experimental and Non-Experimental) to be
used to address the question of whether change observed through monitoring indicators
can be attributed to the project interventions.
g) Set baselines and planning for results -The baseline is the first measurement of an
indicator, which sets the pre-project condition against which change can be tracked and
evaluated.
h) Select data collection methods as applicable.
i) Setting targets and developing a results framework- A target is a specification of the
quantity, quality, timing and location to be realized for a key indicator by a given
date. Starting from the baseline level for an indicator the desired improvement is defined
taking account of planned resource provision and activities, to arrive at a performance
target for that indicator.
j) Plan monitoring, data analysis, communication, and reporting: Monitoring and
Evaluation Plan
k) Implementation monitoring tracking the inputs, activities and outputs in annual or
multiyear work plans, and ‘Results monitoring’ tracking achievement of outcomes and
impact, are both needed. The demands for information at each level of management
need to be established, responsibilities allocated, and plans made for:
i. what data to be collected and when;
ii. how data are collected and analyzed;
iii. who collects and analyses data;
iv. who reports information,
v. when ?
l) Facilitating the necessary conditions and capacities to sustain the M&E System -
organizational structure for M&E, partner’s responsibilities and information
requirements, staffing levels and types, responsibilities and internal linkages,
incentives and training needs, relationships with partners and stakeholders,

Page 15
horizontal and vertical lines of communication and authority, physical resource needs
and budget.

Page 16
 Chapter 7: EVALUATION DESIGN

Developing an evaluation design includes:

 Determining what type of design is required to answer the questions posed
 Selecting a methodological approach and data collection instruments
 Selecting a comparison group
 Sampling
 Determining timing, sequencing, and frequency of data collection

Evaluation research may adopt two general methodological approaches—either a quantitative, a

qualitative or mixed-methods design approach. Quantitative designs normally take the form of
experimental designs. Qualitative evaluation approaches are non-experimental approaches which
answer ‘why’ and ‘how’ questions.

The following are brief descriptions of the most commonly used evaluation (and research) designs.

One-Shot Design In using this design, the evaluator gathers data following an intervention or
program. For example, a survey of participants might be administered
after they complete a workshop.
Retrospective Pre- As with the one-shot design, the evaluator collects data at one time but
test. asks for recall of behaviour or conditions prior to, as well as after, the
intervention or program.
One-Group The evaluator gathers data prior to and following the intervention or
Pre- test-Post- program being evaluated.
test
Design.
Time Series Design The evaluator gathers data prior to, during, and after the implementation
of an intervention or program
Pre-test-Post-test . The evaluator gathers data on two separate groups prior to and
Control-Group following an intervention or program. One group, typically called the
Design experimental or treatment group, receives the intervention. The other
group, called the control group, does not receive the intervention.
Post-test-Only The evaluator collects data from two separate groups following an
Control-Group intervention or program. One group, typically called the experimental or
Design. treatment group, receives the intervention or program, while the other
group, typically called the control group, does not receive the
intervention.
Data are collected from both of these groups only after the intervention.
Case Study Design When evaluations are conducted for the purpose of understanding the
program’s context, participants’ perspectives, the inner dynamics of
situations, and questions related to participants’ experiences, and where
generalization is not a goal, a case study design, with an emphasis on the
collection of qualitative data, might be most appropriate. Case studies
involve in-depth descriptive data collection and analysis of individuals,
groups, systems, processes, or organizations. In particular, the case study
design is most useful when you want to answer how and why questions and
when there is a need to understand the particulars, uniqueness, and
diversity of the case.

Page 17
Decisions for Designing an Evaluation Study

Chapter 8: METHODS OF EVALUATION AND TOOLS

Page 18
 a) Evaluation Methods
Informal and less-structured methods
 Conversation with concerned individuals
 Community interviews
 Field visits
 Reviews of records
 Key informant interviews
 Participant observation
 Focus group interviews

Formal and more-structured methods

 Direct observation
 Questionnaires
 One-time survey
 Panel survey
 Census
 Field experiments

Evaluation Description Remarks

method/Approac
h/Tool
Case study. A detailed description of individuals, Useful in evaluating complex situations and
communities, organizations, events, exploring qualitative impact. Helps to
programmes, time periods or a story illustrate findings and includes comparisons
(commonalities); only when combined
(triangulated) with other case studies or
methods can one draw conclusions about key
principles.
Checklist A list of items used for validating or Allow for systematic review that can be
inspecting whether procedures/steps have useful in setting benchmark standards and
been followed,

Page 19
Evaluation Description Remarks
method/Approac
h/Tool
or the presence of examined behaviours. establishing periodic measures of
improvement.

Community book A community-maintained document of a Where communities have low literacy rates,
project belonging to a community. It can a memory team is identified whose
include written records, pictures, drawings, responsibility it is to relate the written record to
songs or whatever community members feel is the rest of the community in keeping with their
appropriate. oral traditions.

Community A form of public meeting open to all Interaction is between the participants and
interviews/meeti ng. community members. the interviewer, who presides over the
meeting and asks questions following a
prepared interview guide.
Direct A record of what observers see and hear at a An observation guide is often used to reliably
observation specified site, using a detailed observation look for consistent criteria, behaviours, or
form. Observation may be of physical patterns.
surroundings, activities or processes.
Observation is
a good technique for collecting data on
behavioural patterns and physical conditions.
Document A review of documents (secondary data) can It includes written documentation (e.g.
review provide cost-effective and timely project records and reports, administrative
baseline information and a historical databases, training materials,
perspective of the project/programme. correspondence, legislation and policy
documents) as well as videos, electronic data or
photos.
Focus group Focused discussion with a small group A moderator introduces the topic and uses a
discussion. (usually eight to 12 people) of participants to prepared interview guide to lead the
record attitudes, perceptions and beliefs discussion and extract conversation,
relevant to the issues being examined. opinions and reactions.
Interviews. An open-ended (semi-structured) interview Replies can easily be numerically coded for
is a technique for questioning that allows the statistical analysis.
interviewer to probe and pursue topics of
interest in depth (rather than just “yes/no”
questions). A closedended(structured)
interview systematically follows carefully
organized questions (prepared in advance in an
interviewer’s guide) that only allow a limited
range of answers, such as “yes/no” or
expressed by
a rating/number on a scale.
Key informant An interview with a person having special These interviews are generally conducted in
interview. information about a particular topic. an open-ended or semi-structured fashion.
Laboratory Precise measurement of specific objective
testing. phenomenon, e.g. infant weight or water quality
test.

Mini-survey Data collected from interviews with 25 to 50 Structured questionnaires with a limited
individuals, usually selected using number of closed-ended questions are used to
nonprobability generate quantitative data that can be
sampling techniques. collected and analysed quickly.
Most significant A participatory monitoring technique based They give a rich picture of the impact of
change (MSC). on stories about important or significant development work and provide the basis for
changes, rather than indicators. dialogue over key objectives and the value of
development programmes
Participant A technique first used by anthropologists This method gathers insights that might
observation. (those who study humankind); it requires the otherwise be overlooked, but is time-
researcher to spend considerable time (days) consuming.
with the group being studied and to interact
with them as a participant in their
community.
Participatory This uses community engagement techniques It is usually done quickly and intensively –
rapid (or rural) to understand community views on a over a two- to three-week period. Methods
appraisal (PRA). particular issue. include interviews, focus groups and
community mapping. Tools include
stakeholder analysis, participatory rural

Page 20
Evaluation Description Remarks
method/Approac
h/Tool
appraisal, beneficiary assessment, and
participatory monitoring and evaluation.
Questionnaire A data collection instrument containing a set Typically used in a survey
of questions organized in a systematic way, as
well as a set of instructions for the data
collector/interviewer about how to ask the
questions
Rapid appraisal A quick, cost-effective technique to gather This technique shares many of the
(or assessment). data systematically for decision-making, characteristics of participatory appraisal (such
using quantitative and qualitative methods, as triangulation and multidisciplinary teams)
such as site visits, observations and sample and recognizes that indigenous knowledge is
surveys. a critical consideration for decision-making.
Methods include: key informant interview,
focus group discussion, community group
interview, direct
obsevation, and mini-survey
Statistical data A review of population censuses, research
review. A studies and other sources of statistical data.
Story An account or recital of an event or a series of A learning story focuses on the lessons
events. A success story illustrates impact by learned through an individual’s positive and
detailing an individual’s positive experiences negative experiences (if any) with a
in his or her own words. project/programme.
Formal Survey Systematic collection of information from a Includes multi-topic or single topic house
defined population, usually by means of hold/living standards survey, client
interviews or questionnaires administered to a satisfaction surveys, core welfare indicators
sample of units in the population (e.g. person, questionnaire.
beneficiaries and adults). An enumerated Public expenditure tracking surveys- tracking
survey is one in which the survey is flow of public funds and the extent to which
administered by someone trained (a data resources actually reach the target groups.
collector/enumerator) to record responses Sampling-related methods- sample frame,
from respondents. A self- administered sample size, sample method e.g. random –
survey is a written survey completed by the simple (and systematic) or stratified
respondent, either in a group setting or in a Non-random- purposive (and cluster) and
separate location. quota sampling, etc
Respondents must be literate.
Visual Participants develop maps, diagrams, This technique is especially effective where
techniques. calendars, timelines and other visual displays to verbal methods can be problematic due to low-
examine the study topics. Participants can be literate or mixed-language target populations,
prompted to construct visual responses to or in situations where the desired information
questions posed by the interviewers; e.g. by is not easily expressed in either words or
constructing a map of their local area. numbers.
Cost Benefit and Assesses whether or not the costs of an Cost Benefit- measures both inputs and
Cost Effectiveness activity can be justified by the outcomes and outputs in monetary terms
Analysis impacts Cost Effectiveness- inputs in monetary and
outputs in non-monetary terms

M&E Tool/Method Advantages Disadvantages

Survey Good for gathering descriptive Self-report may lead to biased
data reporting
Can cover a wide range of topics Data may provide a general
Are relatively inexpensive to use Can picture but lack depth
be analyzed using a variety of May not provide adequate
existing software information on context

Case studies Provide a rich picture of what is Require a sophisticated and well-
happening, as seen through the trained data collection and
eyes of many individuals reporting team
Allow a thorough exploration of Can be costly in terms of the
interactions between treatment demands on time and resources
and contextual factors Individual cases may be over
Can help explain changes or interpreted or overgeneralized
facilitating factors that might

Page 21
 otherwise not emerge from the
data

Interviews Usually yield richest data, details, Expensive and time consuming
new insights Need well-qualified, highly
Permit face-to-face contact with trained interviewers
respondents Interviewee may distort
Provide opportunity to explore information through recall error,
topics in depth selective perceptions, desire to
Allow interviewer to experience please interviewer
the affective as well as cognitive Flexibility can result in
aspects of responses Allow inconsistencies across interviews
interviewer to explain or help Volume of information very large;
clarify questions, may be difficult to
increasing the likelihood of useful transcribe and reduce data
responses
Allow interviewer to be flexible in
administering interview to
particular individuals or in
particular circumstances

b) PARTICIPATORY M&E
Participatory evaluation is a partnership approach to evaluation in which stakeholders actively engage
in developing the evaluation and all phases of its implementation. Participatory evaluations often
use rapid appraisal techniques. Name a few of them.
 Key Informant Interviews - Interviews with a small number of individuals who are most
knowledgeable about an issue.
 Focus Groups - A small group (8-12) is asked to openly discuss ideas, issues and
experiences.
 Mini-surveys - A small number of people (25-50) is asked a limited number of
questions.
 Neighbourhood Mapping - Pictures show location and types of changes in an area to be
evaluated.
 Flow Diagrams - A visual diagram shows proposed and completed changes in systems.
 Photographs - Photos capture changes in communities that have occurred over time.
 Oral Histories and Stories - Stories capture progress by focusing on one person’s or
organization’s account of change.

E.g. Specific applications of the focus group method in evaluations.

 Identifying and defining problems in project implementation
 Pretesting topics or idea
 Identifying project strengths, weaknesses, and recommendations
 Assisting with interpretation of quantitative findings
 Obtaining perceptions of project outcomes and impacts
 Generating new ideas

Chapter 9: DATA ANALYSIS AND REPORTING

The term “data” refers to raw, unprocessed information while “information,” or “strategic
information,” usually refers to processed data or data presented in some sort of context.
 Data –primary or secondary- is a term given to raw facts or figures before they have been
processed and analysed.
 Information refers to data that has been processed and analysed for reporting and use.
 Data analysis is the process of converting collected (raw) data into usable information.

Page 22
 (i) Quantitative and Qualitative data
 Quantitative data measures and explains what is being studied with numbers (e.g. counts,
ratios, percentages, proportions, average scores, etc).
 Qualitative data explains what is being studied with words (documented observations,
representative case descriptions, perceptions, opinions of value, etc).
 Quantitative methods tend to use structured approaches (e.g. coded responses to surveys)
which provide precise data that can be statistically analysed and replicated (copied) for
comparison.
 Qualitative methods use semi-structured techniques (e.g. observations and interviews) to
provide in-depth understanding of attitudes, beliefs, motives and behaviours. They tend to be
more participatory and reflective in practice.

Quantitative data is often considered more objective and less biased than qualitative data but recent
debates have concluded that both quantitative and qualitative methods have subjective (biased) and
objective (unbiased) characteristics.

Therefore, a mixed-methods approach is often recommended that can utilize the advantages of both,
measuring what happened with quantitative data and examining how and why it happened with
qualitative data.

(ii) Some Data Quality Issues in Monitoring and Evaluation

 Coverage: Will the data cover all of the elements of interest?
 Completeness: Is there a complete set of data for each element of interest?
 Accuracy: Have the instruments been tested to ensure validity and reliability of the data?
 Frequency: Are the data collected as frequently as needed?
 Reporting schedule: Do the available data reflect the periods of interest?
 Accessibility: Are the data needed collectable/retrievable?
 Power: Is the sample size big enough to provide a stable estimate or detect change?

(iii) Data Analysis

Quantitative or qualitative research methods or a complementary combination of both
approaches are used.
Analysis may include:
 Content or textual analysis, making inferences by objectively and systematically
identifying specified characteristics of messages.
 Statistical descriptive techniques, the most common include: graphical description
(histograms, scatter-grams, bar chart,…); tabular description (frequency distribution, cross
tabs,…); parametric description (mean, median, mode, standard deviation, skew- ness,
kurtosis, …).
 Statistical inferential techniques which involve generalizing from a sample to the whole
population and testing hypothesis. Hypothesis are stated in mathematical or statistical terms and
tested through two or one-tailed tests (t-test, chi-square, Pearson correlation, F- statistic, …)

Chapter 10: TERMS OR REFERENCE IN M&E AND EVLUATION REPORT TEMPLATE

(i) Terms of Reference in Evaluation

Evaluation organizers are usually the ones who are in charge of a particular project and want to have
the project evaluated to better manage project operation. Responsibility in the evaluation organizers
differs from those of evaluators, who are usually consultants contracted for the evaluation.

Tasks of the evaluation organizers include:

Page 23
  Preparing the TOR; TOR is a written document presenting the purpose and scope of the
evaluation, the methods to be used, the standard against which performance is to be
assessed or analyses are to be conducted, the resources and time allocated, and reporting
requirements. TOR also defines the expertise and tasks required of a contractor as an evaluator,
and serves as job descriptions for the evaluator.
 Appointing evaluator(s);
 Securing budget for evaluation;
 monitoring the evaluation work;
 Providing comments on the draft;
 publicizing the evaluation report, and
 Providing feedback from the results to concerned parties.

The role of evaluator includes:

 Preparing the detailed evaluation design;
 Collecting and analyzing information, and
 Preparing an evaluation report.

The role of Management includes:

 Management response
 Action on recommendations
 Tracking status of implementation of recommendations

Management Response Template

Prepared by:
Reviewed by:
Evaluation recommendation 1.
Management response:
Key action(s) Time Frame Responsible Tracking*
unit(s) Comments Status
1.1
1.2
Evaluation recommendation 2.
Management response:
Key action(s) Time Frame Responsible Tracking*
unit(s) Comments
Comments Status
2.1
2.2

ii) Evaluation Report Template

The is no single universal format for M&E but t he template is intended to serve as a guide for preparing
meaningful, useful and credible evaluation reports that meet quality standards. It only suggests the content that
should be included in a quality evaluation report but does not purport to prescribe a definitive section- by-section
format that all evaluation reports should follow.

E.g.
Formal reports developed by evaluators typically include six major sections:
(1) Background
(2) Evaluation study questions
(3) Evaluation procedures
(4) Data analyses
(5) Findings
(6) Conclusions (and recommendations)

Or detailed:
Summary sections
A. Abstract
B. Executive summary
II. Background

Page 24
 A. Problems or needs addressed
B. Literature review
C. Stakeholders and their information needs
D. Participants
E. Project’s objectives
F. Activities and components
G. Location and planned longevity of the project
H. Resources used to implement the project
I. Project’s expected measurable outcomes
J. Constraints
III. Evaluation study questions
A. Questions addressed by the study
B. Questions that could not be addressed by the study (when relevant)
IV. Evaluation procedures
A. Sample
1. Selection procedures
2. Representativeness of the sample
3. Use of comparison or control groups, if applicable
B. Data collection
1. Methods
2. Instruments
C. Summary matrix
1. Evaluation questions
2. Variables
3. Data gathering approaches
4. Respondents
5. Data collection schedule
V. Findings
A. Results of the analyses organized by study question
VI. Conclusions
A. Broad-based, summative statements
B. Recommendations, when applicable

0r

Table of contents
Executive summary
 Introduction
 Evaluation scope, focus and approach
 Project facts
 Findings, Lessons Learned
o Findings
o Lessons learned
 Conclusions and recommendations
o Conclusions
o Recommendations
 Annexes/appendices

Or as per organizational requirements (Modified from UNDP, 2009, Handbook on Planning,

Monitoring and Evaluating for Development Results)
The report should also include the following:

1. Title and opening pages—Should provide the following basic information:

 Name of the evaluation intervention
 Time frame of the evaluation and date of the report

Page 25
  Country/Organization/Entity of the evaluation intervention
 Names and organizations of evaluators
 Name of the organization commissioning the evaluation
 Acknowledgements

2. Table of contents
 Should always include lists of boxes, figures, tables and annexes with page references.
3. List of acronyms and abbreviations
4. Executive summary
A stand-alone section of two to three pages that should:
 Briefly describe the intervention (the project(s), programme(s), policies or other
interventions) that was evaluated.
 Explain the purpose and objectives of the evaluation, including the audience for the
evaluation and the intended uses.
 Describe key aspect of the evaluation approach and methods.
 Summarize principle findings, conclusions, and recommendations.
5. Introduction
Should:
 Explain why the evaluation was conducted (the purpose), why the intervention is being
evaluated at this point in time, and why it addressed the questions it did.
 Identify the primary audience or users of the evaluation, what they wanted to learn from the
evaluation and why, and how they are expected to use the evaluation results.
 Identify the intervention (the project(s) programme(s), policies or other interventions) that was
evaluated—see upcoming section on intervention.
 Acquaint the reader with the structure and contents of the report and how the information
contained in the report will meet the purposes of the evaluation and satisfy the information
needs of the report’s intended users.
6. Description of the intervention/project/process/programme —Provide the basis for report users to
understand the logic and assess the merits of the evaluation methodology and understand the
applicability of the evaluation results. The description needs to provide sufficient detail for the report
user to derive meaning from the evaluation. The description should:
 Describe what is being evaluated, who seeks to benefit, and the problem or issue it seeks to
address.
 Explain the expected results map or results framework, implementation strategies, and the key
assumptions underlying the strategy.
 Link the intervention to national priorities, Development partner priorities, corporate strategic
plan goals, or other project, programme, organizational, or country specific plans and goals.
 Identify the phase in the implementation of the intervention and any significant changes (e.g.,
plans, strategies, logical frameworks) that have occurred over time, and explain the
implications of those changes for the evaluation.
 Identify and describe the key partners involved in the implementation and their roles.
 Describe the scale of the intervention, such as the number of components (e.g., phases of a
project) and the size of the target population for each component.
 Indicate the total resources, including human resources and budgets.
 Describe the context of the social, political, economic and institutional factors, and the
geographical landscape within which the intervention operates and explain the effects
(challenges and opportunities) those factors present for its implementation and outcomes.
 Point out design weaknesses (e.g., intervention logic) or other implementation constraints (e.g.,
resource limitations).
7. Evaluation scope and objectives - The report should provide a clear explanation of the evaluation’s
scope, primary objectives and main questions.
 Evaluation scope—The report should define the parameters of the evaluation, for example, the
time period, the segments of the target population included, the geographic area included,
and which components, outputs or outcomes were and were not assessed.
 Evaluation objectives—The report should spell out the types of decisions evaluation users will
make, the issues they will need to consider in making those decisions, and what the
evaluation will need to achieve to contribute to those decisions.

Page 26
  Evaluation criteria—The report should define the evaluation criteria or performance
standards used. The report should explain the rationale for selecting the particular criteria used
in the evaluation.
 Evaluation questions—Evaluation questions define the information that the evaluation will
generate. The report should detail the main evaluation questions addressed by the
evaluation and explain how the answers to these questions address the information needs of
users.
8. Evaluation approach and methods - The evaluation report should describe in detail the selected
methodological approaches, theoretical models, methods and analysis; the rationale for their selection;
and how, within the constraints of time and money, the approaches and methods employed yielded
data that helped answer the evaluation questions and achieved the evaluation purposes. The description
should help the report users judge the merits of the methods used in the evaluation and the credibility
of the findings, conclusions and recommendations.

The description on methodology should include discussion of each of the following:

 Data sources—The sources of information (documents reviewed and stakeholders), the rationale
for their selection and how the information obtained addressed the evaluation questions.
 Sample and sampling frame—If a sample was used: the sample size and characteristics; the
sample selection criteria (e.g., single women, under 45); the process for selecting the
sample (e.g., random, purposive); if applicable, how comparison and treatment groups were
assigned; and the extent to which the sample is representative of the entire target population,
including discussion of the limitations of the sample for generalizing results.
 Data collection procedures and instruments— Methods or procedures used to collect data,
including discussion of data collection instruments (e.g., interview protocols), their
appropriateness for the data source and evidence of their reliability and validity.
 Performance standards/indicators—The standard or measure that will be used to evaluate
performance relative to the evaluation questions (e.g., national or regional indicators, rating
scales).
 Stakeholder engagement—Stakeholders’ engagement in the evaluation and how the level of
involvement contributed to the credibility of the evaluation and the results.
 Ethical considerations—The measures taken to protect the rights and confidentiality of
informants
 Background information on evaluators—The composition of the evaluation team, the
background and skills of team members and the appropriateness of the technical skill mix,
gender balance and geographical representation for the evaluation.
 Major limitations of the methodology—Major limitations of the methodology should be
identified and openly discussed as to their implications for evaluation, as well as steps taken to
mitigate those limitations.
9. Data analysis—The report should describe the procedures used to analyse the data collected to answer
the evaluation questions. It should detail the various steps and stages of analysis that were carried out,
including the steps to confirm the accuracy of data and the results. The report also should discuss the
appropriateness of the analysis to the evaluation questions. Potential weaknesses in the data analysis and
gaps or limitations of the data should be discussed, including their possible influence on the way
findings may be interpreted and conclusions drawn.
10. Findings and conclusions—The report should present the evaluation findings based on the analysis and
conclusions drawn from the findings.
 Findings—Should be presented as statements of fact that are based on analysis of the data. They
should be structured around the evaluation criteria and questions so that report users can readily
make the connection between what was asked and what was found. Variances between planned
and actual results should be explained, as well as factors affecting the achievement of intended
results. Assumptions or risks in the project or programme design that subsequently affected
implementation should be discussed.
 Conclusions—Should be comprehensive and balanced, and highlight the strengths,
weaknesses and outcomes of the intervention. They should be well substantiated by the
evidence and logically connected to evaluation findings. They should respond to key
evaluation questions and provide insights into the identification of and/or solutions to important
problems or issues pertinent to the decision making of intended users.
11. Recommendations—The report should provide practical, feasible recommendations directed to the
intended users of the report about what actions to take or decisions to make. The recommendations
should be specifically supported by the evidence and linked to the findings and conclusions around

Page 27
 key questions addressed by the evaluation. They should address sustainability of the initiative and
comment on the adequacy of the project exit strategy, if applicable.
12. Lessons learned—As appropriate, the report should include discussion of lessons learned from the
evaluation, that is, new knowledge gained from the particular circumstance (intervention, context
outcomes, even about evaluation methods) that are applicable to a similar context. Lessons should be
concise and based on specific evidence presented in the report.
13. Report annexes—Suggested annexes should include the following to provide the report user with
supplemental background and methodological details that enhance the credibility of the report:
 ToR for the evaluation
 Additional methodology-related documentation, such as the evaluation matrix and data
collection instruments (questionnaires, interview guides, observation protocols, etc.) as
appropriate
 List of individuals or groups interviewed or consulted and sites visited
 List of supporting documents reviewed
 Project or programme results map or results framework
 Summary tables of findings, such as tables displaying progress towards outputs, targets, and
goals relative to established indicators
 Short biographies of the evaluators and justification of team composition
 Code of conduct signed by evaluators

Page 28