Evaluating Social Program1

Evaluating social program
Hayet Ben Said

Lesson plan
 Assessing Local Needs and Resources
 SWOT Analysis
 HOW TO EVALUATE
 Theory of Change (ToC) and Measurement
 Identifying an Appropriate Impact Evaluation
Method
 Cost-Effectiveness Analysis
Module I
Assessing Local Needs and Resources

Assessing Local Needs and Resources
• This module provides a guide for developing

and implementing a plan to assess the needs
of communities and the resources available to
them.
Plan
 WHAT DO WE MEAN BY NEEDS AND RESOURCES?
 WHY DEVELOP A PLAN FOR ASSESSING LOCAL
NEEDS AND RESOURCES?
 WHO SHOULD BE INVOLVED IN DEVELOPING A PLAN
FOR ASSESSING LOCAL NEEDS AND RESOURCES?
 WHEN SHOULD NEEDS AND ASSETS BE IDENTIFIED?
 HOW DO YOU DEVELOP A PLAN FOR ASSESSING
LOCAL NEEDS AND RESOURCES?
WHAT DO WE MEAN BY NEEDS AND RESOURCES?
• Needs can be defined as the gap between

what is and what should be. A need can be felt
by an individual, a group, or an entire
community. It can be as concrete as the need
for food and water or as abstract as improved
community cohesiveness. Examining
situations closely helps uncover what is truly
needed, and leads toward future
improvement.
• Resources, or assets are : individuals, organizations

and institutions, buildings, landscapes, equipment --
anything that can be used to improve the quality of
life.
• Examples: mother who volunteers to organize
games and sports for neighborhood children after
school, the farmers' cooperative that makes it
possible for farmers to buy seed and fertilizer
cheaply and to send their produce directly to market
without a middle man,
the bike and walking path where city residents

can exercise -- all represent resources that
enhance community life.
• Every individual is a potential community

asset, and everyone has assets that can be
used for community building.
WHY DEVELOP A PLAN FOR ASSESSING LOCAL NEEDS AND RESOURCES?
• Two questions
First: Why assess needs and resources?
- a deeper understanding of the community (needs ,
resources, assets, culture, social sturctures, history,
etc) allows understanding how to address the
community’s needs and use its resources;
- Assessments encourage community members to
consider the community's assets and how to use
them, as well as the community's needs and how to
address them.
WHY DEVELOP A PLAN FOR ASSESSING LOCAL NEEDS AND RESOURCES?
They will learn how to use their own resources to solve

problems and improve community life.
- It will help understanding what are the issues and
how they arose. By taking advantage of community
resources increases the community's capacity for
solving its own problems and creating its own
change.
- Identifying needs and resources before starting a
program means that you know from the beginning
what you're dealing with so you minimize risks
WHY DEVELOP A PLAN FOR ASSESSING LOCAL NEEDS
AND RESOURCES?
The second question is: Why develop a plan for

that assessment?
- It allows you to involve community members
from the very beginning of the process. This
encourages both trust in the process and
community buy-in and support, not only of
the assessment, but of whatever actions are
taken as a result of it.
AND RESOURCES?
community participation also promotes leadership

from within the community and gives voice to
those who may feel they have none.
- An assessment is a great opportunity to use
community-based participatory research
- A good plan will provide an easy-to-follow road
map for conducting an accurate assessment. So
you will save time and effort in carrying out the
process.
AND RESOURCES?
- A planning process will give community

members the opportunity to voice their
opinions, hopes, and fears about the
community. Their idea of priorities might be
different from those of professionals, but they
shouldn't be ignored.
AND RESOURCES?
• SO
It may be important to address the community's priorities
first, in order to establish trust and show respect, even if
you don't believe that those priorities are in fact the most
important issues.
Building relationships and credibility may be more important
at the beginning of a long association than immediately
tackling what seems to be the most pressing need.
Among other things, community members' priorities may be
the right ones: they may see underlying factors that you
don't yet understand.
WHO SHOULD BE INVOLVED IN DEVELOPING A PLAN FOR ASSESSING LOCAL NEEDS AND
RESOURCES?
• The assessment process benefits greatly when

there's full participation from community
stakeholders. Among those who should be
involved?
- It's both fair and logical to involve those who
are most directly affected by adverse
conditions by those who are experiencing
needs.
WHO SHOULD BE INVOLVED IN DEVELOPING A PLAN
- Health and human service providers. These

individuals and organizations, especially those
that are community-based, often have both a
deep understanding of the community and a
strong empathic connection with the
populations they serve. They can be helpful
both by sharing their knowledge and by
recruiting people from marginalized
populations to contribute to the assessment.
- Government officials. Elected and appointed

officials are often those who can help or
hinder a community change effort. Engaging
them in planning and carrying out an
assessment helps to ensure that they will take
the effort seriously and work to make it
successful.
- Influential people. These can include individuals

who are identified as leaders because of their
positions -- college presidents, directors of
hospitals and other major organizations,
corporate CEOs -- because of the prestige of
their professions -- doctors, professors, judges,
clergy -- or because they are known to be
people of intelligence, integrity, and good will
who care about the community.
- People whose jobs or lives could be affected by

the eventual actions taken as a result of the
assessment. These include teachers, police,
emergency room personnel, landlords, and
others who might have to react if new
community policies or procedures are put in
place.
- Community activists. People who have been

involved in addressing policy or issues that
could come up in the course of the
assessment have a stake in planning the
assessment as well.
WHEN SHOULD NEEDS AND ASSETS BE IDENTIFIED?
ASSESSMENTS OF RESOURCES AND NEEDS SHOULD BE DONE

REGULARLY THROUGHOUT YOUR INITIATIVE:
- Prior to planning the initiative. This gives coalition members,
community leaders, and those being served an idea of how
to improve their circumstances;
- During implementation of an initiative. It is important to
make sure that you are on target not only at the beginning
and the end of a project, but also during its implementation.
Identifying needs and assets during the life of the initiative
helps you use your own resources well, and ensures that
you're addressing the right issues in the right way
WHEN SHOULD NEEDS AND ASSETS BE IDENTIFIED?
- On an ongoing basis. During monitoring and

evaluation, either ongoing or after the
completion of a project, it is important to
celebrate successes and to learn from
setbacks to further community development.
HOW DO YOU DEVELOP A PLAN FOR ASSESSING LOCAL NEEDS AND RESOURCES?
- Recruit a planning group that represents all

stakeholders and mirrors the diversity of the
community
- Design an evaluation process for the assessment,
including the development of the plan;
- Decide why you want to conduct the assessment;
- Determine what data is already available;
- Figure out what other information you need
- Decide what methods you'll use for gathering
information
HOW DO YOU DEVELOP A PLAN FOR ASSESSING LOCAL
- Decide what methods you'll use for gathering

information (Using existing data, listening
sessions and public forums,
Interviews and focus groups, direct, and
sometimes participant, observation, Surveys);
- Decide whom you'll gather information from: the
greater the variety of people that supply your data,
the better perspective you'll have on the real nature,
needs, and resources of the community.
HOW DO YOU DEVELOP A PLAN FOR ASSESSING LOCAL NEEDS AND RESOURCES?
- If you're concerned with domestic violence,

you'd certainly want to include those directly
or indirectly exposed to it, as well as
emergency room personnel and police, in your
data gathering. If you're concerned with
preserving open space, you might look to
include both environmentalists and
developers.

HOW DO YOU DEVELOP A PLAN FOR ASSESSING LOCAL
- Decide who will collect data;

- Decide how you'll reach your informants;
- Decide who will analyze the data and how they'll do
it;
- Plan whatever training is needed;
- Decide how you'll record the results of the
assessment and present them to the community.
Module 2
SWOT Analysis
conducting a SWOT Analysis to identify situational
strengths and weaknesses, as well as opportunities
and threats.
to assess the needs of communities

and the resources available to them.
Plan
 WHAT IS A SWOT ANALYSIS AND WHY SHOULD
YOU USE ONE?
 WHEN DO YOU USE SWOT?
 WHAT ARE THE ELEMENTS OF A SWOT
ANALYSIS?
 HOW DO YOU CREATE A SWOT ANALYSIS?
 HOW DO YOU USE YOUR SWOT ANALYSIS?
WHAT IS A SWOT ANALYSIS AND WHY SHOULD YOU USE ONE?
SWOT stands
for: Strength, Weakness, Opportunity, Threat.
A SWOT analysis guides you to identify your
organization’s strengths and weaknesses (S-
W), as well as broader opportunities and
threats (O-T). Developing a fuller awareness of
the situation helps with both strategic
planning and decision-making.
WHAT IS A SWOT ANALYSIS AND WHY SHOULD YOU USE ONE?
The SWOT method was originally developed for

business and industry, but it is equally useful
in the work of community health and
development, education, and even for
personal growth.
WHAT IS A SWOT ANALYSIS AND WHY SHOULD YOU
USE ONE?
• SWOT is not the only assessment technique

you can use. Compare it with
other assessment tools in the Community Tool
Box
to determine if this is the right approach for
your situation. The strengths of this method
are its simplicity and application to a variety of
levels of operation.
WHEN DO YOU USE SWOT?
• SWOT also offers a simple way of

communicating about your initiative or
program and an excellent way to organize
information you've gathered from studies or
surveys.
WHEN DO YOU USE SWOT?
 You might use it to:

• Explore possibilities for new efforts or solutions to problems.
• Make decisions about the best path for your initiative.
Identifying your opportunities for success in context of
threats to success can clarify directions and choices.
• Determine where change is possible. If you are at a juncture
or turning point, an inventory of your strengths and
weaknesses can reveal priorities as well as possibilities.
• Adjust and refine plans mid-course. A new opportunity might
open wider avenues, while a new threat could close a path
that once existed.
WHAT ARE THE ELEMENTS OF A SWOT ANALYSIS?
 A SWOT analysis focuses on Strengths,

Weaknesses, Opportunities, and Threats.
 The purpose of performing a SWOT is to reveal
positive forces that work together and
potential problems that need to be recognized
and possibly addressed.
 LISTING YOUR INTERNAL FACTORS: STRENGTHS AND

WEAKNESSES (S, W)
• Internal factors include your resources and experiences.
General areas to consider:
• Human resources - staff, volunteers, board members, target
population
• Physical resources - your location, building, equipment
• Financial - grants, funding agencies, other sources of income
• Activities and processes - programs you run, systems you employ
• Past experiences - building blocks for learning and success, your
reputation in the community
 LISTING EXTERNAL FACTORS: OPPORTUNITIES

AND THREATS (O, T)
• Forces and facts that your group does not
control include:
• Future trends in your field or the culture
• The economy - local, national, or international
• Funding sources - foundations, donors,
legislatures
• Demographics - changes in the age, race, gender,

culture of those you serve or in your area
• The physical environment (Is your building in a
growing part of town? Is the bus company
cutting routes?)
• Legislation (Do new federal requirements make
your job harder...or easier?)
• Local, national or international events
HOW DO YOU CREATE A SWOT ANALYSIS?
 WHO DEVELOPS THE SWOT?

• The most common users of a SWOT analysis
are team members and project managers who
are responsible for decision-making and
strategic planning.
• One staff member, or volunteer or stakeholder
may have information about an opportunity or
threat that is essential to understanding your
position and determining your future.
 WHEN AND WHERE DO YOU DEVELOP A SWOT ANALYSIS?

• A SWOT analysis is often created during a retreat or
planning session that allows several hours for
brainstorming and analysis. The best results come when
the process is collaborative and inclusive.
• When creating the analysis, people are asked to pool their
individual and shared knowledge and experience. The
more relaxed, friendly and constructive the setting, the
more truthful, comprehensive, insightful, and useful your
analysis will be.
 HOW DO YOU DEVELOP A SWOT ANALYSIS?

• Steps for conducting a SWOT analysis:
• Designate a leader or group facilitator who has good
listening and group process skills, and who can keep
things moving and on track.
• Designate a recorder to back up the leader if your
group is large. Use newsprint on a flip chart or a large
board to record the analysis and discussion points. You
can record later in a more polished fashion to share
with stakeholders and to update.
• Introduce the SWOT method and its purpose

in your organization. This can be as simple as
asking, "Where are we, where can we go?" If
you have time, you could run through a quick
example based on a shared experience or
well-known public issue.
HOW DO YOU USE YOUR SWOT ANALYSIS?
 Better understanding the factors affecting

your initiative put you in a better position for
action. This understanding helps as you:
• Identify the issues or problems you intend to
change
• Set or reaffirm goals
• Create an action plan
CONCLUSION
 A realistic recognition of the weaknesses and
threats that exist for your effort is the first
step to countering them with a robust set of
strategies that build upon strengths and
opportunities. A SWOT analysis identifies your
strengths, weaknesses, opportunities and
threats to assist you in making strategic plans
and decisions.
Swot matrix
Module III
HOW TO EVALUATE
Outline
 Causal Inference and Counterfactuals
 Randomized Selection Methods
 Diff erence-in-Diff erences
 Matching
 Combining Methods
Causal Inference
 Assessing the impact of a program on a series
of out-comes is equivalent to assessing the
causal effect of the program on those outcomes
 Most policy questions involve cause-and-eff ect
relationships: Does teacher training improve
students’ test scores? Do conditional cash
transfer programs cause better health
outcomes in children? Do vocational training
programs increase trainees’ incomes?
Causal Inference
 To establish causality between a program and
an out-come, we use impact evaluation
methods to rule out the possibility that any
factors other than the program of interest
explain the observed impact.
 What is the impact or causal effect of a
program P on an outcome of interest Y? is
given by the basic impact evaluation formula:
α = (Y | P = 1) − (Y | P = 0).
Causal Inference
(α): the causal impact (α) of a program (P) ;
(Y) is the outcome.
Example : if P denotes a vocational training program
and Y denotes income, then the causal impact of the
vocational training program (α) is the difference
between a person’s income (Y) after participating in
the vocational training program (in other words, when
P = 1) and the same person’s income (Y) at the same
point in time if he or she had not participated in the
program (in other words, when P = 0).
Causal Inference
We would like to measure income at the same point in
time for the same unit of observation (a person, in this
case), but in two different states of the world.
By comparing the same individual with herself at the
same moment, we would have managed to eliminate
any outside factors that might also have explained the
difference in outcomes. We could then be confident
that the relation-ship between the vocational training
program and income is causal.
Causal Inference
So the impact (α) of a program as the difference
in outcomes (Y) for the same individual with
and without participation in a program. Yet we
know that measuring the same person in two
different states at the same time is impossible.
 the counterfactual is what the outcome (Y)
would have been in the absence of a program
(P)
Counterfactual
 The question is how to estimate the
counterfactual?
 Answer : use comparison groups or “control
groups”
How to estimate counterfactual
 Solving the counterfactual problem requires the
evaluator to identify a “perfect clone” for each program
participant
 Key goal of an impact evaluation is to identify a group of
program participants (the treatment group) and a group
of nonparticipants (the comparison group) that are
statistically identical in the absence of the program. If the
two groups are identical, excepting only that one group
participates in the program and the other does not, then
we can be sure that any difference in outcomes must be
due to the program.
• For example, the average age in the treatment group
should be the same as the average age in the
comparison group. Second, the treatment and
comparison groups should react to the program in
the same way. For example, the incomes of units in
the treatment group should be as likely to benefit
from training as the incomes of the comparison
group. Third, the treatment and comparison groups
cannot be differentially exposed to other
interventions during the evaluation period.
 An invalid comparison group is one that differs
from the treatment group in some way other
than the absence of the treatment. Those
additional differences can cause our impact
estimate to be invalid or, in statistical terms,
biased: it will not estimate the true impact of the
program. Rather, it will estimate the effect of the
program mixed with the effect of those other
differences
 How to construct valid comparison groups that
will allow you to estimate the counterfactual?
 First method : before-and-after, or pre-post,
comparisons that compare the outcomes of
program participants prior to and subsequent to
the introduction of a program;
 Second method: with-and-without comparisons
between units that choose to enroll and
units that choose not to enroll.
Comparing Before and After
 A before-and-after comparison attempts to establish

the impact of a program by tracking changes in
outcomes for program participants over time
Module
Theory of Change (ToC) and

Measurement
Theory of Change
 A theory of change is a description of how an
intervention is supposed to deliver the desired
results.
 It describes the causal logic of how and why a
particular project, program, or policy will reach its intended
outcomes.
 It depicts a sequence of events leading to outcomes;
it explores the conditions and assumptions needed for the
change to take place, make explicit the causal logic behind the
program, and map the program interventions along logical
causal pathways.
Why theory of change (objectives)
 Theory of change can:

 help disentangle the inputs and activities that go
into providing the program interventions, the
outputs that are delivered, and the outcomes
that stem from expected behavioral changes
among beneficiaries.
 sets out the causal logic of how and why a
particular project, program, or policy will reach its
intended outcomes (hypothesis)
When developing a theory of change for a
program?
 The best time to develop a theory of change
for a program is at the beginning of the
design process, when stakeholders can be
brought together to develop a common vision
for the program, its goals, and the path to
achieving those goals. Stakeholders can then
start program implementation from a
common understanding of the program, how
it works, and its objectives.
How is theory of change developed?
 A program theory can be developed by

program staff, by an external evaluator, by
program designers, or collaboratively with the
community.
How is theory of change represented?
 It is represented in a logic model:

 Results chain logic models are most appropriate when all the
activities are at the beginning of the process, and less useful
when there are a series of activities throughout participants’
passage through a programme.
 Logframes have a very restrictive format, with only 4 levels. Doing
separate logframes for different components can improve their
usefulness.
 Realist matrices only focus on one step in the chain. While they
provide better insight into this step, there either need to be a
series of them or they need to be used along with other types of
logic models to cover the entire process.
Results chain
 Results chain model is the simplest and
clearest model to outline the theory of
change in the operational context of
development programs.
 It establishes the causal logic from the
initiation of the project, beginning with
resources available, to the end,looking at long-
term goals.
Results chain
 Components:
 Inputs: Resources at the disposal of the project,
including staff and budget;
 Activities: Actions taken or work performed to
convert inputs into outputs;
 Outputs: The tangible goods and services that the
project activities produce (They are directly under the
control of the implementing agency.)
Results chain
 Outcomes: Results likely to be achieved once the

beneficiary population uses the project outputs (They
are usually achieved in the short-to-medium term);
 Final outcomes: The final project goals (They can be
influenced by multiple factors and are typically
achieved over a longer period of time.)
Results chain
 The three main parts of Results chain:

 Implementation: Planned work delivered by the
project, including inputs, activities, and outputs.
These are the areas that the implementation agency
can directly monitor to measure the project’s
performance.
Results chain
• Results: Intended results consist of the outcomes and

final outcomes, which are not under the direct
control of the project and are contingent on
behavioral changes by program beneficiaries. In
other words, they depend on the interactions
between the supply side (implementation) and the
demand side (beneficiaries). These are the areas
subject to impact evaluation to measure
effectiveness.
Results chain
 Assumptions and risks: These are not depicted in

figure. They include any evidence from the literature
on the proposed causal logic and the assumptions on
which it relies, references to similar programs’
performance, and a mention of risks that may affect
the realization of intended results and any mitigation
strategy put in place to manage those risks.
Results chain
Conclusion
 Results chain are useful for all projects, regardless of

whether or not they will include an impact
evaluation, because they allow policy makers and
program managers to make program goals explicit,
thus helping them to understand the causal logic and
sequence of events behind a program.
 Results chains also facilitate discussions around
monitoring and evaluation by making evident what
information needs to be monitored and what
outcome changes need to be included when the
project is evaluated.
Exercice 1
What is a Results Chain of the ministry of education

program?
A ministry is thinking of introducing a new approach to
teaching mathematics in high school. Here some
informations which can be used to draw the Results
chain:
- the number of teachers trained, the number of text
books delivered to classrooms, and the adaptation of
standardized tests to the new curriculum,
Exercice 1
- Staff from the ministry, high school teachers, a

budget for the new math program, and the municipal
facilities where the math teachers will be trained.
- Teachers’ use of the new methods and textbooks in
their classrooms and their application of the new
tests.
Exercice1
Designing the new mathematics curriculum;

developing a teacher training programme;
training the teachers; and commissioning,
printing, and distributing new textbooks,
- Increased high school completion rates and
higher employment rates and earnings for
graduates long,
- Improvements in student performance on the
standardized mathematics tests.
Solution
Exercice 2
Primary Education Management

• Ministry of Education implemented programme to
see whether top-down approach is effective in
improving teacher performance because the
education quality remains poor
Exercice 2
 For that purpuse :

 Officials receive tools and information ,
 Teacher performance improves,
 Intensity and frequency of monitoring increases,
 Learning outcomes improve
 Top-down monitoring programme
Solution
Imputs Activities Outputs Final

Outcomes
Outcomes
Officials Intensity and

Top-down Teacher Learning
receive frequency of
monitoring performance outcomes
tools and monitoring
programme improves improve
information increases
Measurement
 To meseare we need Hypothesis
Indicators
Hypothese for the impact evaluation
What: Assumptions about how these changes might

happen, as a check on whether the activities and outputs
are appropriate for influencing change in the desired
direction in this context.
When? Once you have outlined the results chain

How ? In the high school mathematics example, the
hypotheses to be tested could be the following:
The new curriculum is superior to the old one in imparting
knowledge of mathematics.
Hypothese for the impact evaluation
 Trained teachers use the new curriculum in a more
effective way than other teachers.
 If we train the teachers and distribute the textbooks, then
the teachers will use the new textbooks and curriculum in
class, and the students will follow the curriculum.
 If we train the teachers and distribute the textbooks, then
the math test results will improve by 5 points on average.
• Performance in high school mathematics influences
completion rates and labor market performance.
Risks
• What are the key factors that could diminish the

potential effects of our project, and what steps can
be taken to mitigate them? In any project there are
factors that we cannot control that will affect the
success of our intervention. These could include such
factors as weather, political stability, the local
security situation, and support from local
stakeholders. A good understanding of these factors
is essential for project design, and also for
evaluation.
• Hypothese for the impact evaluation
Imputs Activities Outputs Final

Outcomes
Outcomes
Hypothese 11
Hypothese Hypothese 2 Hypothese 3 Hypothese 4 Hypothese 5
Indicators
 Quantitative and qualitative

 SMART
 Specific :to measure the information required as
closely as possible
 Measurable: to ensure that the information can be
readily obtained
 Attributable: to ensure that each measure is linked
to the project’s efforts
 Realistic: to ensure that the data can be obtained in
a timely fashion, with reasonable frequency, and at
reasonable cost
Indicators
 Indicators used both to monitor program

implementation and to evaluate results. It is useful to
engage program stakeholders in selecting these
indicators, to ensure that the ones selected are good
measures of program performance.
 Define several indicators for each element in the
results chain, especially regarding outcomes or
higher-level outcomes (one to three key indicators
that best reflect each element in the results chain).
Theory of Change
Needs assessment
Data to collect
Research Imput
questions Indicators
Hypothese
Output Indicators
Hypothese
Outcomes Indicators
Hypothese
Final outcomes Indicators
Example: Theory of ChangeTop-Down
Example
Case study
Given the lack of employment opportunities combined

with low levels of skills and barriers to starting a
business, the Youth Opportunities Program decided
to focus on a comprehensive entrepreneurship
program that would provide vocational skills training,
cash grants, and other support services.
Case study
Questions: based on these informations

1) Establishing the results chain
2) Identify key indicators and assumptions
Informations:
- Lower youth unemployment, Higher household
income, Reduction in poverty, Improvement in self-
sufficiency;
- New curriculum approved,Teachers trained in new
curriculum, Youth trained in basic and technical skills,
Youth participated in simulations
Case study
- Budget Staff, Local counterparts,Trainers,

Partnerships Facilities, Equipment, Supplies,
Technical expertise
- Re-design curriculum to include business skills,
Organize teachers workshops, Provide remedial
education, Provide technical training,Provide
financial literacy training
Case study
HIGHER-LEVEL
INPUTS ACTIVITIES OUTPUTS OUTCOMES
What the Products Long-

Resources program or
mobilized New term
does services effects
Budget curriculum
Staff
Re-design approved Lower youth
Local curriculum to Teachers unemployment
counterparts include business trained in Higher
Trainers skills new household
Partnerships Organize teachers curriculum income
Facilities workshops Youth trained Reduction in
Equipment Provide remedial in basic and poverty
Supplies education technical Improvement
Technical Provide technical skills in self-
expertise training Youth sufficiency
Provide financial participated
in simulations
Indicators
Category Sample Target Example of Indicators
Input Two trainers and facility - Two trainers skilled,
within budget of equipped and deployed
US$10,000 - Cost of program in U.S.
dollars within desired
budget
Activity Provide life skills training - Number of training hours
for youth (20 hours) delivered
- Number of youth
participating by age,
gender, level of education
- Date by which training
was provided
Outputs 100 youth participated in Number of youth who
training finished the training (by
age, gender, level of
education)
Indicators
Outcomes By the end of the program:

- Number and percentage
of youth able to express
ideas clearly measured
against a predetermined
test score card
of youth with improved
verbal and nonverbal
communication skills
measured against a
predetermined test score
card
of youth who report
feeling comfortable
approaching employers
Higher-Level Outcomes Increased household By 2015, average monthly
income household income
increased by 20%
compared to baseline
Example of assumptions for a school-based entrepreneurship program

Category Assumption
Outputs (New curriculum approved Teachers willing to be trained
Teachers trained Youth trained in business youth can attend training
skills
Outcome (Curriculum widely - Curriculum accepted by local school
implemented authorities
Better understanding of business - Better quality of teaching lead to the fact
Improved soft skills that youth can attend school regularly
Improved employability
Increased interest for higher education
Higher-Level outcomes (Lower youth New skills are demanded and rewarded
unemployment Higher household by labor market
income)
ModuleV
Choosing the right type of evaluation

Plan
 What Is the Purpose of the Evaluation?
 Linking Evaluation Questions to Evaluation
Design
 Does Our Operational Context Fit the Desired
Type of Evaluation?
 Types of Programs That Usually Justify an
Impact Evaluation
What Is the Purpose of the Evaluation?
(what we want to get out of the evaluation)
 Why this question: what we want to get out of the
evaluation?
Because the answer to this question allow us to decide if an
evaluation is necessary and which design should be
chosen;
What decision will be taken by the evaluation and what
kinds of information are needed to make that decision? Do
we want to know more about how well our programs are
being implemented, whether our programs are meeting
their objectives, or whether our beneficiaries are actually
better off as a result of our intervention?
 So?
 As a program managers and evaluators, we
first: establish our questions and learning
objectives
Second: select the most appropriate evaluation
tool to provide the necessary information.
 Evaluations address three types of questions:
First: Descriptive questions: describe processes, conditions,
organizational relationships, and stakeholder views.
Second: Normative questions: compare what is
taking place to what should be taking place. They
compare the current situation with the specific
objectives and targets that have been defined
(Has our project been implemented as intended?
Is it performing as expected?).
Third: Cause-and-effect questions : examine

outcomes and try to measure what difference an
intervention makes. They ask whether objectives
have been achieved as a result of our project
(What is the impact or causal effect of our
program on outcomes of interest?).
Examples of evaluation questions

Inputs Activities Outputs Outcomes Higher-

Level
Outcomes
Descriptive How does the cost of Do youth How many Are Is local
the program compare know about youth participants youth
to similar the program participate satisfied unemploy-
interventions? and how they (by age, with the ment rising
What are the qualify to sex, etc.)? program? or falling?
qualifications of service join? Who drops How many Are
providers? What delivery out? program household
What are other ongoing mechanisms What participants incomes
interventions? are being services find evolving?
used? are used employme
the most? nt within 3
months?
Level
Outcomes
What Is the Purpose of the
Evaluation?
Normative Do we spend Is the process Do we achieve Does Are more
as much as we for selecting the desired participant house-
have participants fair gender balance income holds
budgeted? and equitable? in participants? increase by becoming
Are the staff Is the program Will we reach 20%, as self-
and financial implementatio the goal of planned? sufficient?
resources n delayed? training 5,000 Do 80% of Are more
adequate? youth per year? beneficiarie house-
s find a job holds
within 3 reaching
months food
of security?
graduation,
as
required?
Inputs Activities Outputs Outcomes Higher-Level
Outcomes
Cause-and- - - - Does the program Does the project

Effect affect boys and girls contribute to
differently? reducing poverty
As a result of the job in the area?
training, do What other
participants have impacts does this
higher paying jobs intervention have
than they otherwise on the living
would have? conditions of the
wider
community?
What Is the Purpose of the
Evaluation?
 The connection between evaluation questions and
evaluation criteria.
 Evaluations are periodic assessments
of the relevance, efficiency, effectiveness, impact, and
sustainability of our intervention (OECD 1991).
 Normative questions (relevance, efficiency, and
effectiveness
 Causality (impact)
 Sustainability can be either normative or cause-and-
effect
Criteria Description Details Type of
• Evaluation
Question
Relevance Do the objectives To what extent are the Normative
match the objectives of the
problems or needs program still valid?
that are being Are the activities and
addressed? outputs of the program
consistent with the
overall attainment of its
objectives?
Efficiency Is the project Is the program or Normative

delivered in a project implemented in
timely and cost- the most efficient way?
effective manner? What are the costs per
output/beneficiary and
how do these compare
with similar
interventions?
Description Details Type of Evaluation
Question
Effectiveness To what extent does To what extent were Normative
the intervention the intended results
achieve its objectives? achieved?
What are the major
factors influencing the
achievement or
nonachievement of the
objectives?
Impact What are the positive What are the higher- Cause and effect
and negative changes level outcomes
produced by the resulting from the
intervention? program or project?
What real difference
has the activity made to
the beneficiaries?
Description Details Type of Evaluation
Question
Sustainability Are there lasting To what extent do Normative or
benefits after the the benefits of a cause-and-effect
intervention is project continue
completed? after donor funding
ceases?
What are the major
factors that
influence the
achievement or non
achievement of
sustainability?
 Each of these three kinds of questions—descriptive,
normative, and cause-and- effect—leads to different
considerations for the type of evaluation to be set
up.
 Program managers and evaluators can allocate a
potential question into one of the three types and
then consider the implications of each type of
question for the development of an evaluation
design.
 Thus, by choosing a set of evaluation questions we
define the menu of appropriate monitoring and
evaluation tools that will allow answering them.
Cause-and-effect
Linking Evaluation Questions to Evaluation

Design
Type of evaluation questions
to be answered

Descriptive Cause-and-
Normative
(only) effect
No evaluation Performance Impact

(monitoring evaluation evaluation
only)
Process
evaluation
Cost-effectiveness/Cost-
Design
 No Evaluation: if a program manager requires only
descriptive information about the intervention, for
example, because the project is in a very early stage
and the objective is to obtain some general
information about how the program is being
implemented.
 Performance evaluations: assess how well program
objectives have been formulated as well as the
program’s progress in achieving these objectives.
Design
Such evaluations can be carried out across all stages of
implementation, but they are particularly common
for mid-term reviews (when their focus is on learning
for program management) or at program completion
(when their focus is on account- ability and lessons
learned for future interventions).
Performance evaluations may incorporate more
extensive data collection, such as a before-and-after
comparison of participant outcomes or additional
qualitative tools.
Design
 Process Evaluation: allows to fully understanding
how
a program works and seek to assess how well a
program is being implemented. it determines
whether there are gaps between planned and
realized activities and outputs and try to understand
the reasons for gaps. Building on descriptive
information such as what activities are being offered
and who is participating in the program (or who is
not).
Design
Process evaluations tend to rely on a mix of
quantitative and qualitative tools, including key
informant interviews, user satisfaction surveys, direct
observation, and focus groups.
 Impact evaluations are the only type of evaluation to
specifically answer cause-and-effect questions in a
quantifiable manner. Such questions require us to
determine not only whether the desired outcomes
occurred but also whether those outcomes occurred
because the program was implemented.
Design
To estimate the causal effect of a program on outcomes
of interest, any method chosen must estimate the so-
called counterfactual, that is, what would have
happened to program participants in the absence of
the program. To do this, impact evaluations require
finding a comparison group; This is what makes
impact evaluations different from all other
evaluations. As a result, they tend to require more
time and quantitative skills, and they typically cost
more than other evaluation types.
Design
 Cost-Effectiveness and Cost-Benefit Analyses: assess
monetary and nonmonetary program costs and
compare them with alternative uses of the same
resources and the benefits produced by the
intervention (Baker 2000). Cost-effectiveness analysis
(CEA) measures the cost per output or outcome (e.g.,
$300 per youth trained, $500per job created) and
compares this cost to similar interventions of our
own and other organizations.
•
Design
It thus answers the question about how much output or
outcome we get per dollar spent (descriptive) and whether
there is a gap with our expectations (normative). Cost-benefit
analysis (CBA), in turn, weighs the total expected costs against
the total expected benefits (outcomes) of an intervention,
where both costs and benefits are typically expressed in
monetary terms. For instance, if our program were to help 500
youth find and keep jobs or set up sustainable small
businesses, we would estimate the aggregate benefits in terms
of higher incomes, better health, lower crime, etc., and
compare these benefits to the overall costs of the
intervention.
Does Our Operational Context Fit the Desired
Type of Evaluation?
After formulating the right questions and identifying a
potential type of evaluation, we need to assess the
operational context of the intervention to
understand what evaluation can be implemented
within the given constraints.
Timing
Questions about what kind of information is needed
are closely related to the question of when the
results of the evaluation need to be available.
Knowing when they need to be available determines
when the information needs to be collected.
Does Our Operational Context Fit the Desired
Type of Evaluation?
At What Stage of the Program Is the Information Needed?
How Long Does the Evaluation Take? Information needs vary
depending on the program lifecycle. For example, a program that
has just been planned may require a cost-effectiveness analysis to
help determine whether or not to implement the program. How
long an evaluation takes partly depends on the methods used for
collecting and analyzing data, which differ according to the type
of evaluation, and on the breadth and depth desired for the
particular study, which differ within each type of evaluation

At what point in a program’s lifecycle different evaluation strategies are best conducted?
Small pilot Full pilot Full launch
-Resolve
operational issues, -Run fully
-Establish people Expand the program if
operational program Phase 2 succeeds
interest in with limited
program, Incorporate lessons
geographical scope from impact
-Verify that - Conduct an impact evaluation
participants are evaluation to analyze
satisfied with the Continue monitoring
the effect of the and performance
progam program evaluations
Types of Programs That Usually Justify an Impact
Evaluation
 Strategically relevant and influential;
 Innovative or untested;
 Replicable.
Overview of main evaluation types
Cost-
Performance Process Evaluation Impact Effectiveness
Evaluation Evaluation and Cost-
Benefit
•
Analyses
•
What are Do programs Are adequate resources How have Are program
have clear and systems participants’ costs justified
the main objectives? (management, well-being compared
questions Is the program information, etc.) in changed as a with similar
answered design place? result of the interventions?
appropriate to Is the program being intervention? Are aggregate
by this type achieve the implemented according Are there any program costs
of objectives? to design? unintended justified in
evaluation? To what extent What are the actual consequences terms benefits
have program steps and activities , positive or achieved?
objectives been involved in delivering a negative, on
achieved? product or service? program
Do priorities need What do beneficiaries or participants?
to be changed? other stakeholders know
or think about the
program?
Evaluation Evaluation and Cost-Bene
Analyses
When can It may be It may be It should be Conducted during

his conducted at conducted designed during an ex ante
the planning of a analysis to
evaluation early stages of at any program, but the determine
be implementation, time. final results will whether the
conducted for mid-term typically not be program is worth
? review, or at available till after implementing, or
program the program after the program
(phase) has been is completed to
completion completed determine the fin
costs
How long 1–3 months 1–6 6months(retrosp 1–3 months

does it months ective evaluation)
12–24 months
ake? (prospective
evaluation)
Performance Process Impact Cost-
Evaluation Evaluation Evaluation Effectiveness and
Cost-Benefit
Analyses
What data Desk review of A mix of Statistical and Desk review of

existing interviews with econometric existing program
collection documents and program staff analysis of survey documents and
and analyses selected field and and relevant
are visits, possibly clients, user administrative literature as well
complemented satisfaction data, ideally as key informant
required? by monitoring surveys, record combined with interviews
data analysis, review, direct qualitative data
beneficiary and observation, analysis
stakeholder focus groups, and
interviews, mini- analysis of
surveys, focus monitoring data
groups, etc.
Who carries Usually Internal or Independent Independent

evaluation team,
out the independent independent including evaluator
evaluator
evaluation? evaluator lead evaluator,
field coordinator,
survey firm
Performance Process Impact Cost-
Evaluation Evaluation Evaluation Effectiveness and
Cost-Benefit
Analyses
What skills Program Process Statistical Valuation

are needed? analysis, analysis, and and
possibly quantitative econometric economic
simple and analysis, analysis of
quantitative qualitative possibly program
methods methods qualitative costs and
methods benefits
What Every Every Programs that Every program

programs program program are:
Innovative and
are best untested
suited for Strategically
this relevant and
evaluation? influential
Replicable
Key Points
1. Our learning objectives are the point of departure for

any evaluation. This requires formulating evaluation
questions across all levels of the results chain. In
general, evaluation questions can be descriptive,
normative, or cause-and-effect;
2. The choice of the evaluation strategy depends on the
evaluation questions. Purely descriptive information
needs may not require an evaluation, and monitoring
may suffice. Normative questions are most
commonly answered through process or performance
evaluations. Cause-and-effect questions need impact
evaluations. Cost-effectiveness and cost-benefit
analyses answer whether the costs involved in an
intervention are justifiable.
Key Points
3. Only impact evaluations allow us to prove

whether a program has been successful and
to generate knowledge that can potentially
be generalized beyond the intervention itself;
4. Choosing an appropriate type of evaluation
depends on the operational context. It is
therefore crucial to understand whether the
costs in terms of money, staff, and time for
each evaluation are appropriate for a given
intervention.
Key Points
5. Since impact evaluations tend to be the most

resource intensive type of evaluation, they
should be applied selectively to answer
strategic questions or to assess innovative
pilot interventions.
Module VI
Identifying an Appropriate Impact

Evaluation Method
Introduction
 The choice of an impact evaluation method will

depend not only on the theoretical quality of the
method, but also on the operational context of the
program. Program managers therefore need to be
involved during the evaluation design to make sure
the evaluation responds to the needs and context
of the intervention.
 the impact evaluation method to be used depends
on timing, coverage, targeting, and resources.
131
Impact evaluations
 Impact evaluations try to answer whether an

intervention (the cause) improves outcomes among
beneficiaries (the effect).
 For example: Does our vocational training program
increase trainees’ incomes?
Does our start-up mentoring program foster business
creation and sustainability?

132
Impact evaluations
• Establishing causality between intervention activities

and the outcomes we observe can be complicated
because other factors may also influence the
outcomes we are interested in. For instance, simply
observing that business creation increased after our
entrepreneurship program was implemented is not
proof of our program’s success because other factors
such as local economic conditions or regulations
about starting a business may have improved during
the life of our program and contributed to business
creation.
133
Impact evaluations
 The purpose of impact evaluations is precisely to

overcome this challenge by measuring to what
extent a particular program, and only that program,
contributed to the change in the outcomes of
interest?
134
what is impact?
Impact is the change in outcomes that can be

directly attributed to our program and nothing
else.
Outcome with the program
– Outcome in the absence of the program
= Impact
135
What is counterfactual?
Counterfactual is referring to what would have

happened to the beneficiary if the program
had not taken place.
136
Illustration of a program impact

Outcome
Outcome
Without intervention
Without intervention
137
Time Time
What is counterfactual?
It is impossible to know what their situation

would have been in the absence of the
program. That is, we cannot know with
certainty what would have happened to them
if they had not participated in our program. As
a result, we will never be able to get the real
counterfactual, so an estimate must suffice.
138
How to estimate impact?
In order to estimate what would have happened to

beneficiaries in the absence of the program
(counterfactuals), we construct comparison groups
that share as many characteristics with the
beneficiaries (treatment group) as possible. If a good
comparison group (non treatment group) can be
identified, comparing outcomes between the
comparison group and the beneficiaries yields the
impact of the program.
139
Two types of impact evaluation techniques to find valid

comparison groups.
First: experimental techniques randomly separate the
eligible population into those who receive the program
and those who don’t.
Second: quasi-experimental techniques try to find a valid
comparison group among nonparticipants, mirroring
the treatment group as closely as possible.

140
Random assignment

EXPERIMENTAL
QUASI-EXPERIMENTAL
Eligible Eligible
Random
assignement
Eligible non
Treatment Comparison Treatment
participants
Comparison
141
Key Points
1. The impact of a program is the change in

outcomes that can be directly attributed to the
intervention. Understanding impact requires
that we isolate the effects of the program from
other factors influencing beneficiary outcomes.
2. Measuring program impact requires a
counterfactual, knowing what would have
happened to our program participants in the
absence of the intervention.
142
Key Points
3. In order to estimate what would have

happened to beneficiaries in the absence of
the program, we construct comparison groups
that share as many characteristics with the
beneficiaries as possible. If a good comparison
group can be identified, comparing outcomes
between the comparison group and the
beneficiaries (treatment group) yields the
143
Key Points
4. Impact evaluation techniques to find valid

comparison groups can be classified as one of
two types. Experimental techniques randomly
separate the eligible population into those
who receive the program and those who
don’t. Quasi-experimental techniques try to
find a valid comparison group among
nonparticipants, mirroring the treatment
group as closely as possible.
144
Method 1 lottery design
• Definition: A lottery is a simple and transparent way

to assign people to groups who will receive our
services (the treatment group) and those who won’t
(the comparison group). It is a statistical regularity
that if a large enough sample of people from the
same population of interest are randomly assigned to
one of two groups, then both groups will, on average,
have similar observable characteristics (age, gender,
height, level of education, and the like) and
unobservable characteristics (such as motivation and
state of mind).
145
Method 1 lottery design
Through randomization, the difference in outcomes

we observe between the two groups at the end
of our program can be attributed to the
intervention because all other factors that could
influence the outcomes are, on average, equal.
Lottery designs are considered the most robust type
of impact evaluation, so the results are usually
the most trusted by donors, stakeholders, and
governments.
146
How It Works
Total population
Step 1: Define eligible population Ineligible
Select sample
Step 2:
Step 3: Random assignment
Treatment Comparison
147
How It Works
 Step 1: Define the Eligible Population

The first step in a randomized controlled trial is to find a
group of eligible young people for a program. What is
important here is to have very clear and transparent
criteria (age, gender, income level, employment status,
etc.) and to be able to communicate who will be
eligible to join the program and who won’t.

148
How It Works
 Step 2: Select a Sample for the Evaluation

To evaluate an intervention, we do not need to
test everyone who will participate in the
intervention. We just need to choose a
representative group of people that is
numerous enough for the purpose of our
evaluation; this is called our sample.
149
How It Works
 Choosing the sample for the evaluation can be

done in two ways, depending on whether the
program is large or small. A small program may find
that there are 10,000 eligible beneficiaries, such as
urban street youth aged 16–24. The program may
have the budget to help 500 of them. Ideally, a
comparison group will be equal in size to the
treatment group, so 1,000 out of the 10,000 street
youth will need to be selected for the program and
evaluation.
150
How It Works
• Large programs may be bigger than the sample size

needed for an evaluation. If the program is able to
serve 4,000 youth, it is not necessary to find an
additional 4,000 youth for comparison. Instead, only
1,000 may be needed. The program can then
identify a sample of 5,000 youth from the total
population of 10,000. Of these, 3,000 youth can be
guaranteed admission to the program. The
remaining 2,000 will then be randomly split
between the program and the comparison group.
151
SMALL PROGRAM (500 How It Works
LARGE PROGRAM (4,000)
SMALL PROGRAM (500 YOUTH)
YOUTH)Eligible population
Eligible population (10,000) (10,000)
Sample (1,000) Sample (5,000)
Sample (1,000) Program randomly Sub-sample for

guaranteed (3,000)
evaluation (2,000)
Treatment (500) Comparison (500)
Random assignment
Comparison
Treatment (1,000) (1.000)
152
How It Works
In order to make the selection representative of the total eligible

population of 10,000 street youth, the sample (whether 1,000
in the first case or 5,000 in the second case) should be selected
at random from the eligible population. By selecting randomly,
the program participants will, on average, have similar
characteristics as the total eligible population. Even though we
include only a limited number of youth in the study, the
potential impact of the program can be generalized to the
entire eligible population, in this case, 10,000 youth.

153
How It Works
Step 3: Randomize Assignment

The next step is to assign the selected sample of youth to
treatment and comparison groups roughly equal in size.
Randomization can be via traditional techniques such as
flipping a coin, rolling dice, or drawing names out of a hat.
Randomization can be done publicly, if desired, if the
sample is relatively small (drawing 2,000 names out of a
hat, for example, would not be very practical). Alternatively
—and more appropriately if the number of people is large
—we can randomize by using computer software, such as
MS Excel.
154
How It Works
• By assigning our sample to treatment or comparison

groups randomly, we select participants fairly, and
we also develop a good counterfactual: if the sample
size is big enough, youth in the treatment group
have, on average, the same observable and
unobservable characteristics as those in the
comparison group.
155
How It Works
 Randomization can be conducted at the individual,

group, or community level, according to program
needs.
 Individual level. Individual randomization is best
for programs in which outcomes will be measured
for each participant. There may be problems with
this method, such as spillover, which occurs when
individuals in the comparison group receive some
of the treatment through informal means.
156
How It Works
 Group level. If there is not a list of people’s

names readily available, or if there is an
expectation that people selected for the
comparison group may receive the program
anyway, then randomizing at a group level
may be better.

157
How It Works
This works particularly well for programs that operate

on a group level, targeting schools, vocational
training centers, youth centers, and the like. In this
case, groups of people are randomized into
treatment or comparison cohorts. Randomization at
the group level can help reduce spillover effects and
may be easier than randomizing on the individual
level. Alternatively, it may also be possible to
randomize at the subgroup level, such as
classrooms in schools.
158
How It Works
 Village/community level. Programs may also

choose to randomize at the level of villages,
neighborhoods, communities, or even
districts, when activities are implemented on
that level, or when spillover effects are
expected to occur beyond the group level.
159
How It Works
For example, if there are 100 villages in a district of

interest and we don’t have the resources to work
with all of them, we may randomly choose to work
with fifty of them, while keeping the other fifty
villages as a comparison. All the youth within the
respective treatment villages would then be eligible
to participate in the program.
160
When Can I Use a Lottery Design?
A randomized lottery evaluation is used when the

evaluation is planned in advance of
implementation (prospective evaluation) and
when the program can serve only a fraction of
eligible youth. Despite the costs associated with
collecting new data, a simple random lottery can
be the cheapest option for an evaluation because
it may require fewer surveys and lower numbers
of respondents.
161
Advantages of a Lottery Design?
 A lottery design is the most robust method for

developing a counterfactual because it leads to a
very well matched comparison group. It is therefore
considered the most credible design to measure
impact;
 It is a simple method. The impact of the
program in a random trial is simply the mean
difference in outcomes between treatment
and comparison groups.
162
Advantages of a Lottery Design?
 It allows for communities to be directly involved in

the selection process for a fair and transparent
allocation of benefits.
163
Disadvantages of a Lottery Design?
 It requires a comparison group to be excluded from

the program for the duration of the impact
evaluation;
 Organizations must ensure that partners and local
stakeholders consent to the method;
 The internal validity of a lottery design depends on
the fact that the randomization works and is
maintained throughout the study, which may not be
easy to do.
164
Method 2: Randomized Phase-In Design
 It is a variation of the lottery design.

 It applies to programs that are rolled out over time .
 Because creating a pure comparison group in which
youth are never given the program is sometimes
impossible. Never giving the program to a group of
needy individuals can be both politically and
programmatically difficult.
165
How It Works
 The main difference between a phase-in design and a lottery

design is the method of assigning people to treatment and
comparison groups.
 When an intervention is delivered in several tranches over
time, a phase-in design gives each eligible person or group the
same chance of receiving the program under each of the
tranches.
 One set of people is then randomly selected to receive the
treatment in the first period, while another group is selected
to receive the program in the second period, a third group in
the third period, and so on. For the time that certain groups
are waitlisted, they can serve as the comparison group until
they receive the program 166
How It Works
Eligible
Random assignment
Group 1 Group 2 Group 3
Year 1
Year 2
Year 3
Treatment Comparison
167
How It Works
It is the case an NGO which may have the

budget to train 1,500 people, but it may not
have the capacity to conduct all of the training
at once. Instead, it chooses to train 500
people per year for three years. If it can
identify all 1,500 participants in the beginning,
a phased-in randomization may be the best
evaluation method for them. The 1,500 youths
are randomly split into three groups.
168
How It Works
In year one, while group 1 receives training, groups 2 and 3

are waitlisted and can serve as the comparison group. In
year two, only group 3 remains for comparison. By year
three, all three groups will have received training.
As individuals are selected at random for the different
groups, it is possible to compare those offered treatment
first with those offered treatment later. However, because
everyone eventually gets the program, the phase-in design
is usually not well suited to finding the long-term impact
of a program because eventually there is no comparison
group.
169
When Can I Use A Phase-in Design?
As with a lottery design, a phase-in evaluation is

prospective and requires excess demand and the
ability to assign participants randomly to treatment
and comparison groups. The phase-in design is better
suited than a lottery design to large programs that
expect to rollout interventions over a number of
years.
170
Advantages of Randomized Phase-In Design
 Phase-in designs produce a robust counterfactual,

have a fair and transparent selection process, and
allow for comparing the impacts of program
alternatives.
 Because everyone eventually receives the program
with this method, phase-in studies can be politically
expedient.
171
Disadvantages of Randomized Phase-In Design
 As with the lottery method, there are challenges to

guaranteeing successful randomization and
maintaining treatment and comparison groups over
time.
 Participants may not wait to join in the program. If
they do, there is a risk that they will change their
behaviors in the meantime and therefore will not be
a comparable comparison group.
 The phase-in method cannot estimate the long-term
172
Method 3: Discontinuity Design
 The reality is that in many cases we are not able to

plan the evaluation during the program design.
In fact, many programs use a continuous ranking of
potential beneficiaries, such as test scores, credit
scores, poverty index, or age, and have a cut off point
for acceptance into the program.
173
Method 3: Discontinuity Design
For example, applicants to a business plan

competition or a microfinance bank may be given
a score based on a set of criteria and assigned a
grade 1–100. If youth score at or above the
minimum threshold (85) and above, they receive
start-up financing. If they score below, they are
not accepted into the program. Eligibility rankings
like these can be used for an impact evaluation.
174
How It Works
• The premise of discontinuity (or eligibility-

index) evaluation designs is that the people
who score just above and just below a defined
threshold are not very different from one
another, or at least the difference may be
continuous across the scores.
175
How It Works
• If we have a situation in which some of those

people who receive the program (those just
above the threshold) and some of those who
don’t (those just below the threshold) are not
fundamentally different from one another,
then comparing the outcomes of these two
groups, in turn, would allow us to analyze
program impact.
176
How It Works
• Figure above illustrates what we may find when analyzing

the impact of a youth microcredit initiative. The left graph
indicates that, at the time of applying to the program,
those who achieved better scores already tended to have
higher incomes. There may be many reasons for this, such
as that those with somewhat better education are already
earning more and that their education also helped them
secure better scores. Or those who are more motivated in
starting a business were already more entrepreneurial,
reflected in higher incomes, and that motivation also
helped them convince the jury to support them.
177
Sample discontinuity chart

178
How It Works
• When starting the program, the local microfinance

bank decided that the threshold to receive a loan was
85. did microcredit program have any impact on
incomes. As illustrated in figure above (right graph), we
assume that those who received a score below 85 have
the same outcomes as previously, while the income of
those with a score of 85 and above increased across
the board. From this information, it is possible to
identify the impact of the program, which will be
represented by the difference in outcomes (that is, the
discontinuity of the linear relationship) near the cutoff.
179
When Can I Use a Discontinuity Design?
The discontinuity design can be used for both

prospective and retrospective evaluations. That is,
unlike the randomized techniques discussed above, it
can also be used when the program is already
underway or completed. For this method to work,
however, we need many observations in the region
immediately above and below the cutoff point in
order to have sufficient numbers of youth that we
can compare with one another. A discontinuity
design requires similar data collection as a lottery
design, and thus has a similar cost.
• 180
Advantages of a Discontinuity Design
 The discontinuity method takes advantage of

existing targeting rules;
 It provides unbiased estimates for
participants near the cutoff.
 It does not require randomization of any kind,
so it may be more politically acceptable than
other methods.
181
Disadvantages of a Discontinuity Design
 The method requires a very specific threshold

for determining groups.
 Impact estimates are valid only for the margin
near the cutoff and cannot be generalized to
people whose scores are further away from the
threshold. The technique does not provide an
average impact for program participants.
 It requires large evaluation samples since only
the observations around the cutoff can be used.
182
Overview of impact evaluation techniques
Methodolo Description Comparison Required Data Needed When to Use?

-gy Group Assumptions
Lottery A sample of Those Randomization Post- If study can be
eligible selected by is successful intervention designed before
individuals is lottery and complied data for the program
randomly with. treatment and begins. If resources
assigned into The two groups comparison are scarce and it is
those who are statistically groups important to
receive the identical on Baseline data ensure that fair
intervention observed and are desirable methods are used
and those unobserved to enroll needy
who do not. factors. people into the
Impact is the program
difference in If the comparison
outcomes group will never
between the get the program for
two groups. the length of the
evaluation
183
Methodology Description Comparison Required Data When to Use?

Group Assumptions Needed
Random Eligible Those Randomization is If study can be

Phase-In individuals waitlisted successful and designed before
are assigned complied with. the program
to Those in the later begins. If resources
treatment program phases are scarce and it is
tranches will not important to
and receive significantly ensure that fair
the program change their methods are used
sequentially. behavior while to enroll needy
waiting to people into the
participate in the program
program. If program is rolled
out over time
184
Methodology Description Comparison Required Data When to Use?
Group Assumptions Needed
Discontinuity Individuals are Those close Those near Post- If

ranked based on to the cutoff either side of intervention randomization
specific, who were the cutoff are data for is not possible
measurable not eligible very similar in those or the
criteria. observed and nearest the evaluation
There is a cutoff unobserved cutoff starts after the
that determines characteristics Baseline program
who is eligible to . data are begins
participate. desirable If the selection
Outcomes of is based on a
participants and continuous
nonparticipants ranking with
close to the cutoff
cutoff line are
then compared.
185
Key Points
 Only a selected range of impact evaluation methods

allow for obtaining a reliable counterfactual and
trustworthy results.
 Lottery designs, randomized phase-in, and
discontinuity- designs all produce estimates of the
counterfactual through explicit program assignment
rules.
 No single method is best for every program. The best
method depends on the operational context (i.e.,
timing, coverage, and targeting) of the program.
186
Key Points
 Whenever possible, it is highly desirable to plan the

impact evaluation before the program is
implemented. Retrospective evaluations tend to be
less robust and may not be possible at all if the
necessary data was not collected through other
channels.
187
Module VI
Cost-Effectiveness Analysis
Why cost-effectiveness analysis?
 Having a realistic estimate of the cost-effectiveness

analysis , in turn, allows us to answer the following
questions:
 How can we choose among alternatives? Which
program is the most cost-efficient given a certain level
of impact?
 Would we be able to scale up? If costs are high, it is
unlikely that we will be able to reach a large number of
beneficiaries.
 Is any intervention always better than none? If the total
costs outweigh the total benefits of the program,
maybe the resources are better spent somewhere else.189
cost-effectiveness analysis (CEA)
and cost-benefit analysis (CBA)
 Cost-benefit analysis also identifies and quantifies the
full cost of a program and further weighs those costs
against the dollar value of all program benefits.
Knowing the net benefits and net costs of the
intervention, it is then possible to calculate the ratio of
benefits to costs and to determine the return to
society on CB=
thetotal
organization’s
benefit/total cost investment.
• For example, the benefits cost ratio is 2 if net benefits

are $1,000 per person and net costs are $500.
190
What is cost-effectiveness analysis (CEA)
 Cost-effectiveness analysis (CEA): summarizes a

complex program in terms of a simple ratio of costs
to impacts :
191
What is cost-effectiveness analysis (CEA)
Cost – effectiveness analysis is a method that

consists of defining the objectives of a project
and choosing the solution that minimizes
discounted capital and recurrent costs for a
given output or maximizes the output for a
given cost.
192
• It is a quantitative method to assess wether
the effect of intervention is justified by its
costs
193
cost-effectiveness analysis (CEA)
and cost-benefit analysis (CBA)
• Both CEA and CBA can be used before the
intervention or during or after the program.
However, only retrospective analysis will provide
practitioners with the full information of actual
costs and benefits to determine the overall success
of the intervention. In fact, an impact evaluation is
a necessary condition for having a reliable estimate
of the program’s direct and indirect benefits (in this
course, we only emphasize on direct benefits).
194
Weighing costs and benefits
COSTS BENEFITS
- All resources that the program uses and

purchases (salaries, materials and Monetary benefits (income & productivity
supplies, rentals, maintenance, travel, gains, lower health care expenditure, etc.)
overhead, etc.) Non-monetary benefits (increased
psychological well-being, empowerment,
- Capital expenses (computers, software, community cohesion, quality of life, etc.)
textbooks, vehicles, etc.)
Multiplier effects to third parties (spillovers
- Cost to third parties (volunteer time,
of skills)
time and transportation costs for
participants, value of inkind donations,
environmental damage to the public, etc.)
195
How to choose between cost-effectiveness
analysis(CEA) and cost-benefit analysis (CBA)
 Trying to put a dollar value on many intangible
benefits may be difficult and subjective, and it can
represent a big challenge, especially for CBA. Hence,
CBA is usually considered most useful when there are
multiple types of benefits and consensus about how
to quantify them in monetary terms.
196
How to Calculate Net Benefits for Participants
Example: say a skills training program is found to

increase income by $100 per person, per year, on
average.
The benefits of a program can be measured by its
impact on individual participants and this is a
significant amount of money, and may represent
great success for the program.
197
 Assume the program costs $1 million to

implement, with $400,000 to conduct the
training and $600,000 in overhead, including all
staff salaries. If it reaches 1,000 people, the
program thus costs $1,000 per person to
implement, with $400 going toward training and
$600 going toward overhead. Is it worth running?
 The answer to this question is based on three
criterias.
198
 First, the program’s impact must equal or exceed the

impact of giving individuals cash equal to the cost of
running the program. In the example above, the
impact must be compared with the effects of giving
each person $1,000 cash. There are two possible
scenarios. First, a person actually uses the $1000 and
purchases training with that money. The cost of the
training is still only $400 per person, which yields the
same $100 per person per year return. The individual
then has an extra $600 to use how they please, and
is thus better off than with the program.
199
In a second scenario, a person may use the money
for something other than training that is less
useful for her over the long term, such as
cigarettes or other nonessential consumer goods.
In the latter case, the program is worth running.
200
Second, the program must have equal or greater return than running
other programs. Is it possible that another program could have
realized the same or greater impact per person for less money? This
question requires a comparison of results across different program
options. The program that has the greatest impact but costs the
least is then the best program to continue running.
201
 Finally, the net present value of the return should be more

than the cost of the program. Present value is a way of
thinking about the value of money today compared with
its value in the future. For example, we take the value of
the money obtained yearly (the $100 per person return on
training) and adjust its value over a period of time
according to the discount rate, which in most cases equals,
the local interest rate. The net present value is simply the
sum of the present value adjusted over a period of time.
This is represented in the table above, using an interest
rate of 20 percent (Discounting :taking account of time).
202
One dollar receivable in one year ’ s time is not

worth as much as a dollar today. Because if a
person had a dollar today, he or she could
invest it to obtain a larger sum than a dollar in
a year ’ s time. Also, whether or not the
money is invested, people prefer to have
things now rather than in the future.
203
How to Calculate Net Benefits for
Participants
Discounting is simply a way of taking account of
time by attaching lower weights to cash flows
(or benefits) which occur in the more distant
future than to those which occur relatively
soon.
The cost – effectiveness ratio will then be the
discounted costs divided by the discounted
physical outputs.
204
Year Present Value ($)

0 100
1 83
2 69
3 58
4 48
5 40
6 33
7 28
8 23
9 19
10 16
Net present value 517
205
In this example, today $100 is valued at $100.

However, at our current interest rate, the $100
of income today will be worth only $16 in ten
years. Over a 10-year period, the net present
value of our $100 is $517. Over the entire
lifespan of a participant, the net present value
will be at most $600. Thus, a return of $100
per person per year works out to a maximum
return of $600 per person over their lifetime.
206
• According to these criteria, in today’s dollars, our

outlay for the program ($1,000) is greater than the
benefit to individuals. So, the value of the training is
not enough to justify the program.
207
Bibligraphy
• Bamberger, M., Rugh, J. et, L. (2012). RealWorld Evaluation : Working Under Budget,
Time, Data, and Political Constraints (2ème éd.). Sage Publications. Chen, H.,
Donaldson, S., Mark, M. (éds.) (2011). Advancing Validity in Outcome Évaluation :
Theory and Practice. New Directions for Evaluation. No130. Jossey-Bass et American
Evaluation Association. Funnell, S. et Rogers, P (2011). Purposeful Program Theory :
Effective Use of Theories of Change and Logic Models. Jossey-Bass Publications.
• Jones, H. (2011). « A guide to monitoring and evaluating policy influence. » Londres :
Overseas Development Institute. http://www.odi.org.uk/ resources/docs/6453.pdf
• Mayne, J. (2001). « Addressing attribution through contribution analysis : Using
performance measures sensibly », Canadian Journal of Program Evaluation, 16(1), 1-
24.
• Mayne, J. (2011). « Contribution Analysis : Advancing Cause and Effect. » In
Evaluating the Complex : Attribution, Contribution, and Beyond, édité par Robert
Schwartz, Mita Mara, et Kim Forss. Londres : Transaction Publishing
Bibligraphy
• OCDE/CAD. (2011). Evaluating Impacts : An Overview. http://www.oecd.org/document/17/0,3
746,en_21571361_34047972_46179985_1_1_1_1,00. html
• OCDE/CAD. (2002). Glossaire des principaux termes relatifs à l’évaluation et la gestion axée sur
les résultats. http://www.oecd.org/dataoecd/29/21/2754804.pdf Malhotra, A. (2010).
« Connecting Routine M&E with Impact Evaluation, » International Center for Research on
Women. Présentation à InterAction/3ie Impact Evaluation Design Clinic, Washington, D.C., avril
2010. http:// www.interaction.org/sites/default/files/18/ Anju_Malhotra_-
_Connecting_Routine_M&E_ with_Impact_Evaluation.pdf
• Paul, S. (2002). Holding the State to Account : Citizen Monitoring in Action. Bangalore : Public
Affairs Centre.
• Pawson, R. et Tilley, N. (1997). Realistic Evaluation. Sage Publications.
• Scriven, M. (1991). Evaluation Thesaurus (4ème éd). Sage Publications.
• Woolcock, M. (2011). « Guest Post on The Importance of Time and Trajectories in
Understanding Project Effectiveness. » Development Impact. Blogs.worldbank.org.
http://blogs.worldbank.org/impactevaluations/ guest-post-michael-woolcock-on-the-
importanceof-time-and-trajectories-in-understanding-project-effe.

Evaluating Social Program1

Uploaded by

Copyright:

Available Formats

Evaluating Social Program1

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Evaluating Social Program1

Uploaded by

Copyright:

Available Formats

Evaluating social program

Hayet Ben Said

Assessing Local Needs and Resources

• This module provides a guide for developing

• Needs can be defined as the gap between

• Resources, or assets are : individuals, organizations

the bike and walking path where city residents

• Every individual is a potential community

They will learn how to use their own resources to solve

The second question is: Why develop a plan for

community participation also promotes leadership

- A planning process will give community

• The assessment process benefits greatly when

- Health and human service providers. These

- Government officials. Elected and appointed

- Influential people. These can include individuals

- People whose jobs or lives could be affected by

- Community activists. People who have been

ASSESSMENTS OF RESOURCES AND NEEDS SHOULD BE DONE

- On an ongoing basis. During monitoring and

- Recruit a planning group that represents all

- Decide what methods you'll use for gathering

- If you're concerned with domestic violence,

- Decide who will collect data;

to assess the needs of communities

The SWOT method was originally developed for

• SWOT is not the only assessment technique

• SWOT also offers a simple way of

 You might use it to:

 A SWOT analysis focuses on Strengths,

 LISTING YOUR INTERNAL FACTORS: STRENGTHS AND

 LISTING EXTERNAL FACTORS: OPPORTUNITIES

• Demographics - changes in the age, race, gender,

 WHO DEVELOPS THE SWOT?

 WHEN AND WHERE DO YOU DEVELOP A SWOT ANALYSIS?

 HOW DO YOU DEVELOP A SWOT ANALYSIS?

• Introduce the SWOT method and its purpose

 Better understanding the factors affecting

 A before-and-after comparison attempts to establish

Theory of Change (ToC) and

 Theory of change can:

 A program theory can be developed by

 It is represented in a logic model:

 Outcomes: Results likely to be achieved once the

 The three main parts of Results chain:

• Results: Intended results consist of the outcomes and

 Assumptions and risks: These are not depicted in

 Results chain are useful for all projects, regardless of

What is a Results Chain of the ministry of education

- Staff from the ministry, high school teachers, a

Designing the new mathematics curriculum;

Primary Education Management

 For that purpuse :

Imputs Activities Outputs Final

Officials Intensity and

 To meseare we need Hypothesis

What: Assumptions about how these changes might

When? Once you have outlined the results chain