Introduction Components of Program Evaluation

Meeting 1: Introduction to Impact Evaluation

Gumilang Aryo Sahadewo

Universitas Gadjah Mada

February 8, 2021
Introduction Components of Program Evaluation

Outline of the Talk

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

Outline of the Talk

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

References

The contents of this lecture are based on:

Impact Evaluation in Practice by Paul Gertler, Sebastian
Martinez, Patrick Premand, Laura Rawlings, Christel
Vermeersch.
What is Evaluation, PPT presentation by Marc Shotland,
J-PAL
Introduction Components of Program Evaluation

Evidence-Based Policy Making

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

Evidence-Based Policy Making

Policy

Policy = Intervention = Program

Introduction Components of Program Evaluation

Evidence-Based Policy Making

What is policy?

Google: a course or principle of action adopted or proposed by

a government, party, business, or individual.
Wikipedia: A policy is a deliberate system of principles to
guide decisions and achieve rational outcomes.
How can we formulate an appropriate policy for a particular
problem?
Introduction Components of Program Evaluation

Evidence-Based Policy Making

What is policy?

Google: a course or principle of action adopted or proposed by

a government, party, business, or individual.
Wikipedia: A policy is a deliberate system of principles to
guide decisions and achieve rational outcomes.
How can we formulate an appropriate policy for a particular
problem?
Introduction Components of Program Evaluation

Evidence-Based Policy Making

What is policy?

Google: a course or principle of action adopted or proposed by

a government, party, business, or individual.
Wikipedia: A policy is a deliberate system of principles to
guide decisions and achieve rational outcomes.
How can we formulate an appropriate policy for a particular
problem?
Introduction Components of Program Evaluation

Evidence-Based Policy Making

How can we improve school participation rate?

The Government of Indonesia aims to improve elementary

school (SD) participation rate. What intervention would you
recommend to the GoI?
Introduction Components of Program Evaluation

Evidence-Based Policy Making

What is the most effective intervention to increase school

participation (enrollment and attendance)?

Textbooks
Free school uniforms
Treat intestinal worms
Merit scholarships
Improve curriculum and teaching
Provide better materials
Increase awareness of returns to education
Introduction Components of Program Evaluation

Evidence-Based Policy Making

Impact evaluation provides an answer to such inquiry

Introduction Components of Program Evaluation

Evidence-Based Policy Making

Impact evaluation provides an answer to such inquiry

Impact evaluation is also a tool to decide on which programs

to invest given a limited amount of resources
One of the tools is cost-effectiveness analysis:
for a given amount of money, how much can you
increase/improve....
There are many solutions to achieve the intended result, but
some are cost-effective than others
Introduction Components of Program Evaluation

What is Impact Evaluation?

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

What is Impact Evaluation?

What is impact evaluation?

Impact evaluation is one of many approaches that support

evidence-based policy.
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations are periodic, objective assessments of a planned,

ongoing, or completed project, program, or policy.
Evaluations are used selectively to answer specific questions
related to:
design
implementation
results
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations are periodic, objective assessments of a planned,

ongoing, or completed project, program, or policy.
Evaluations are used selectively to answer specific questions
related to:
design
implementation
results
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations are periodic, objective assessments of a planned,

ongoing, or completed project, program, or policy.
Evaluations are used selectively to answer specific questions
related to:
design
implementation
results
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations are periodic, objective assessments of a planned,

ongoing, or completed project, program, or policy.
Evaluations are used selectively to answer specific questions
related to:
design
implementation
results
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations are periodic, objective assessments of a planned,

ongoing, or completed project, program, or policy.
Evaluations are used selectively to answer specific questions
related to:
design
implementation
results
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Evaluations

Evaluations can address three types of questions (Imas and

Rist, 2009)
Descriptive questions
ask about what is taking place
processes, conditions, organizational relationship, stakeholder
views
Normative questions
compare what is taking place to what should be taking place
inputs, activities, outputs
Cause-and-effect questions
What difference the intervention makes to outcomes?
Introduction Components of Program Evaluation

What is Impact Evaluation?

Impact Evaluations

Impact evaluations seek to answer a specific cause-and-effect

question:
What is the impact (or causal effect) of a program on an
outcome of interest?
The focus is on the impact.
That is, changes attributable to:
a program
program modality
design innovation
Introduction Components of Program Evaluation

What is Impact Evaluation?

Impact Evaluations

Impact evaluations seek to answer a specific cause-and-effect

question:
What is the impact (or causal effect) of a program on an
outcome of interest?
The focus is on the impact.
That is, changes attributable to:
a program
program modality
design innovation
Introduction Components of Program Evaluation

What is Impact Evaluation?

Impact Evaluations

Impact evaluations seek to answer a specific cause-and-effect

question:
What is the impact (or causal effect) of a program on an
outcome of interest?
The focus is on the impact.
That is, changes attributable to:
a program
program modality
design innovation
Introduction Components of Program Evaluation

What is Impact Evaluation?

Impact Evaluations

Impact evaluations seek to answer a specific cause-and-effect

question:
What is the impact (or causal effect) of a program on an
outcome of interest?
The focus is on the impact.
That is, changes attributable to:
a program
program modality
design innovation
Introduction Components of Program Evaluation

What is Impact Evaluation?

Impact Evaluations

Impact evaluations seek to answer a specific cause-and-effect

question:
What is the impact (or causal effect) of a program on an
outcome of interest?
The focus is on the impact.
That is, changes attributable to:
a program
program modality
design innovation
Introduction Components of Program Evaluation

What is Impact Evaluation?

Impact Evaluations

Impact evaluations seek to answer a specific cause-and-effect

question:
What is the impact (or causal effect) of a program on an
outcome of interest?
The focus is on the impact.
That is, changes attributable to:
a program
program modality
design innovation
Introduction Components of Program Evaluation

What is Impact Evaluation?

Impact Evaluations

Impact evaluations seek to answer a specific cause-and-effect

question:
What is the impact (or causal effect) of a program on an
outcome of interest?
The focus is on the impact.
That is, changes attributable to:
a program
program modality
design innovation
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Prospective versus Retrospective Impact Evaluations

Retrospective evaluation assess program impact after the

program has been implemented
looks for treatment and comparison groups ex post
Prospective evaluation are developed at the same time as the
program is being designed
they are built into program implementation
treatment and comparison (control) groups
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Prospective versus Retrospective Impact Evaluations

Retrospective evaluation assess program impact after the

program has been implemented
looks for treatment and comparison groups ex post
Prospective evaluation are developed at the same time as the
program is being designed
they are built into program implementation
treatment and comparison (control) groups
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Prospective versus Retrospective Impact Evaluations

Retrospective evaluation assess program impact after the

program has been implemented
looks for treatment and comparison groups ex post
Prospective evaluation are developed at the same time as the
program is being designed
they are built into program implementation
treatment and comparison (control) groups
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Prospective versus Retrospective Impact Evaluations

Retrospective evaluation assess program impact after the

program has been implemented
looks for treatment and comparison groups ex post
Prospective evaluation are developed at the same time as the
program is being designed
they are built into program implementation
treatment and comparison (control) groups
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Prospective versus Retrospective Impact Evaluations

Retrospective evaluation assess program impact after the

program has been implemented
looks for treatment and comparison groups ex post
Prospective evaluation are developed at the same time as the
program is being designed
they are built into program implementation
treatment and comparison (control) groups
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Advantages of Prospective Evaluations

Collect baseline data to establish measures of outcomes of

interest before the program has started
ensure treatment and comparison groups are similar
measure pre-intervention outcomes
targeting effectiveness
Focuses both the program and the evaluation on intended
results
design clarifies program objectives
The treatment and comparison groups are identified before
the intervention being evaluated is implemented.
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Advantages of Prospective Evaluations

Collect baseline data to establish measures of outcomes of

interest before the program has started
ensure treatment and comparison groups are similar
measure pre-intervention outcomes
targeting effectiveness
Focuses both the program and the evaluation on intended
results
design clarifies program objectives
The treatment and comparison groups are identified before
the intervention being evaluated is implemented.
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Advantages of Prospective Evaluations

Collect baseline data to establish measures of outcomes of

interest before the program has started
ensure treatment and comparison groups are similar
measure pre-intervention outcomes
targeting effectiveness
Focuses both the program and the evaluation on intended
results
design clarifies program objectives
The treatment and comparison groups are identified before
the intervention being evaluated is implemented.
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Advantages of Prospective Evaluations

Collect baseline data to establish measures of outcomes of

interest before the program has started
ensure treatment and comparison groups are similar
measure pre-intervention outcomes
targeting effectiveness
Focuses both the program and the evaluation on intended
results
design clarifies program objectives
The treatment and comparison groups are identified before
the intervention being evaluated is implemented.
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Advantages of Prospective Evaluations

Collect baseline data to establish measures of outcomes of

interest before the program has started
ensure treatment and comparison groups are similar
measure pre-intervention outcomes
targeting effectiveness
Focuses both the program and the evaluation on intended
results
design clarifies program objectives
The treatment and comparison groups are identified before
the intervention being evaluated is implemented.
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Advantages of Prospective Evaluations

Collect baseline data to establish measures of outcomes of

interest before the program has started
ensure treatment and comparison groups are similar
measure pre-intervention outcomes
targeting effectiveness
Focuses both the program and the evaluation on intended
results
design clarifies program objectives
The treatment and comparison groups are identified before
the intervention being evaluated is implemented.
Introduction Components of Program Evaluation

Prospective versus Retrospective Impact Evaluation

Advantages of Prospective Evaluations

Collect baseline data to establish measures of outcomes of

interest before the program has started
ensure treatment and comparison groups are similar
measure pre-intervention outcomes
targeting effectiveness
Focuses both the program and the evaluation on intended
results
design clarifies program objectives
The treatment and comparison groups are identified before
the intervention being evaluated is implemented.
Introduction Components of Program Evaluation

5 Components of Program Evaluation

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

5 Components of Program Evaluation

5 Components of program evaluation

Introduction Components of Program Evaluation

5 Components of Program Evaluation

What do you think is the most cost-effective way to reduce

diarrhea?

A case study in Kenya:

J-PAL Cleaning Springs in Kenya
Introduction Components of Program Evaluation

5 Components of Program Evaluation

What do you think is the most cost-effective way to reduce

diarrhea?

Developed piped water infrastructure

Improve existing water sources
Increase supply of and demand for chlorine
Education on sanitation and health
Improved cooking stoves for boiling water
Improve sanitation infrastructure
Introduction Components of Program Evaluation

Identifying the Problem

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

Identifying the Problem

Need Assessment

Does the problem we proposing to solve actually exist?

What is the likely source of the problem?
Of the solutions proposed and tried, why are they failing?
Who is in most need?
Introduction Components of Program Evaluation

Identifying the Problem

Need Assessment

Does the problem we proposing to solve actually exist?

What is the likely source of the problem?
Of the solutions proposed and tried, why are they failing?
Who is in most need?
Introduction Components of Program Evaluation

Identifying the Problem

Need Assessment

Does the problem we proposing to solve actually exist?

What is the likely source of the problem?
Of the solutions proposed and tried, why are they failing?
Who is in most need?
Introduction Components of Program Evaluation

Identifying the Problem

Need Assessment

Does the problem we proposing to solve actually exist?

What is the likely source of the problem?
Of the solutions proposed and tried, why are they failing?
Who is in most need?
Introduction Components of Program Evaluation

Identifying the Problem

Need assessment

Does the problem of diarrhea really exist?

Diarrheal disease killed approximately 2.6 million people a year
between 1990 and 2000
20% all child deaths (under 5 years old) are from diarrhea
In non-fatal cases, diarrhea leads to severe dehydration
Introduction Components of Program Evaluation

Identifying the Problem

Need assessment

Does the problem of diarrhea really exist?

Diarrheal disease killed approximately 2.6 million people a year
between 1990 and 2000
20% all child deaths (under 5 years old) are from diarrhea
In non-fatal cases, diarrhea leads to severe dehydration
Introduction Components of Program Evaluation

Identifying the Problem

Need assessment

Does the problem of diarrhea really exist?

Diarrheal disease killed approximately 2.6 million people a year
between 1990 and 2000
20% all child deaths (under 5 years old) are from diarrhea
In non-fatal cases, diarrhea leads to severe dehydration
Introduction Components of Program Evaluation

Identifying the Problem

Need assessment

Does the problem of diarrhea really exist?

Diarrheal disease killed approximately 2.6 million people a year
between 1990 and 2000
20% all child deaths (under 5 years old) are from diarrhea
In non-fatal cases, diarrhea leads to severe dehydration
Introduction Components of Program Evaluation

Identifying the Problem

The problem
Introduction Components of Program Evaluation

Theory of Change

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

Theory of Change

Theory of change

A theory of change is a description of how an intervention is

supposed to deliver the desired results
It is the causal logic of how and why a particular program,
program modality, or design innovation will reach its intended
outcomes
How will the program address the needs put forth in your
needs assessment?
What are the prerequisites to meet the needs?
How and why are those requirements currently lacking or
failing?
How does the program intend to target or circumvent
shortcomings?
What services will be offered?
Introduction Components of Program Evaluation

Theory of Change

Theory of change

A theory of change is a description of how an intervention is

supposed to deliver the desired results
It is the causal logic of how and why a particular program,
program modality, or design innovation will reach its intended
outcomes
How will the program address the needs put forth in your
needs assessment?
What are the prerequisites to meet the needs?
How and why are those requirements currently lacking or
failing?
How does the program intend to target or circumvent
shortcomings?
What services will be offered?
Introduction Components of Program Evaluation

Theory of Change

Theory of change

A theory of change is a description of how an intervention is

supposed to deliver the desired results
It is the causal logic of how and why a particular program,
program modality, or design innovation will reach its intended
outcomes
How will the program address the needs put forth in your
needs assessment?
What are the prerequisites to meet the needs?
How and why are those requirements currently lacking or
failing?
How does the program intend to target or circumvent
shortcomings?
What services will be offered?
Introduction Components of Program Evaluation

Theory of Change

Theory of change

A theory of change is a description of how an intervention is

supposed to deliver the desired results
It is the causal logic of how and why a particular program,
program modality, or design innovation will reach its intended
outcomes
How will the program address the needs put forth in your
needs assessment?
What are the prerequisites to meet the needs?
How and why are those requirements currently lacking or
failing?
How does the program intend to target or circumvent
shortcomings?
What services will be offered?
Introduction Components of Program Evaluation

Theory of Change

Theory of change

A theory of change is a description of how an intervention is

supposed to deliver the desired results
It is the causal logic of how and why a particular program,
program modality, or design innovation will reach its intended
outcomes
How will the program address the needs put forth in your
needs assessment?
What are the prerequisites to meet the needs?
How and why are those requirements currently lacking or
failing?
How does the program intend to target or circumvent
shortcomings?
What services will be offered?
Introduction Components of Program Evaluation

Theory of Change

Theory of change

A theory of change is a description of how an intervention is

supposed to deliver the desired results
It is the causal logic of how and why a particular program,
program modality, or design innovation will reach its intended
outcomes
How will the program address the needs put forth in your
needs assessment?
What are the prerequisites to meet the needs?
How and why are those requirements currently lacking or
failing?
How does the program intend to target or circumvent
shortcomings?
What services will be offered?
Introduction Components of Program Evaluation

Theory of Change

Theory of change

A theory of change is a description of how an intervention is

supposed to deliver the desired results
It is the causal logic of how and why a particular program,
program modality, or design innovation will reach its intended
outcomes
How will the program address the needs put forth in your
needs assessment?
What are the prerequisites to meet the needs?
How and why are those requirements currently lacking or
failing?
How does the program intend to target or circumvent
shortcomings?
What services will be offered?
Introduction Components of Program Evaluation

Theory of Change

What is a potential solution to the diarrhea problem?

Introduction Components of Program Evaluation

Theory of Change

Alternative solution?
Introduction Components of Program Evaluation

Theory of Change

Alternative solution?
Introduction Components of Program Evaluation

Theory of Change

Alternative solution?
Introduction Components of Program Evaluation

Theory of Change

Alternative solution?
Introduction Components of Program Evaluation

Theory of Change

Is contamination really the source of the problem?

Hygiene (Esrey, 1996)

42% live without a toilet at home
Nearly 2.6 billion people lack any improved sanitation facilities
(WHO)
People are more willing to pay for convenient water than clean
water
Chlorine is very cheap, but the demand is less than 10% of
households purchase treatment!
Introduction Components of Program Evaluation

Theory of Change

Is contamination really the source of the problem?

Hygiene (Esrey, 1996)

42% live without a toilet at home
Nearly 2.6 billion people lack any improved sanitation facilities
(WHO)
People are more willing to pay for convenient water than clean
water
Chlorine is very cheap, but the demand is less than 10% of
households purchase treatment!
Introduction Components of Program Evaluation

Theory of Change

Is contamination really the source of the problem?

Hygiene (Esrey, 1996)

42% live without a toilet at home
Nearly 2.6 billion people lack any improved sanitation facilities
(WHO)
People are more willing to pay for convenient water than clean
water
Chlorine is very cheap, but the demand is less than 10% of
households purchase treatment!
Introduction Components of Program Evaluation

Theory of Change

Is contamination really the source of the problem?

Hygiene (Esrey, 1996)

42% live without a toilet at home
Nearly 2.6 billion people lack any improved sanitation facilities
(WHO)
People are more willing to pay for convenient water than clean
water
Chlorine is very cheap, but the demand is less than 10% of
households purchase treatment!
Introduction Components of Program Evaluation

Theory of Change

Is contamination really the source of the problem?

Hygiene (Esrey, 1996)

42% live without a toilet at home
Nearly 2.6 billion people lack any improved sanitation facilities
(WHO)
People are more willing to pay for convenient water than clean
water
Chlorine is very cheap, but the demand is less than 10% of
households purchase treatment!
Introduction Components of Program Evaluation

Theory of Change

Theory of change
Introduction Components of Program Evaluation

Theory of Change

Developing a results chain

How can we achieve the intended result?

If [inputs] and [activities] produce [outputs], they will lead to
[outcomes]
Introduction Components of Program Evaluation

Theory of Change

Developing a results chain

How can we achieve the intended result?

If [inputs] and [activities] produce [outputs], they will lead to
[outcomes]
Introduction Components of Program Evaluation

Theory of Change

Log frame
Introduction Components of Program Evaluation

Theory of Change

Process evaluation

Was the program carried out as planned?

Are basic tasks being completed?
Is the intervention reaching the target population?
Is the intervention being completed well or efficiently and to
the beneficiaries’ satisfaction?
Introduction Components of Program Evaluation

Theory of Change

Process evaluation

Was the program carried out as planned?

Are basic tasks being completed?
Is the intervention reaching the target population?
Is the intervention being completed well or efficiently and to
the beneficiaries’ satisfaction?
Introduction Components of Program Evaluation

Theory of Change

Process evaluation

Was the program carried out as planned?

Are basic tasks being completed?
Is the intervention reaching the target population?
Is the intervention being completed well or efficiently and to
the beneficiaries’ satisfaction?
Introduction Components of Program Evaluation

Theory of Change

Process evaluation

Was the program carried out as planned?

Are basic tasks being completed?
Is the intervention reaching the target population?
Is the intervention being completed well or efficiently and to
the beneficiaries’ satisfaction?
Introduction Components of Program Evaluation

Impact Evaluation

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

Impact Evaluation

Process evaluation

Process evaluations determine if a program is running in the

way it is supposed to run
Impact evaluations determines if a program creates a change
in an outcome(s)?
Did concrete encased springs decrease diarrhea rates?
What was the impact?
66% reduction in source water e coli concentration
24% reduction in household E coli concentration
25% reduction in incidence of diarrhea
Introduction Components of Program Evaluation

Impact Evaluation

Process evaluation

Process evaluations determine if a program is running in the

way it is supposed to run
Impact evaluations determines if a program creates a change
in an outcome(s)?
Did concrete encased springs decrease diarrhea rates?
What was the impact?
66% reduction in source water e coli concentration
24% reduction in household E coli concentration
25% reduction in incidence of diarrhea
Introduction Components of Program Evaluation

Impact Evaluation

Process evaluation

Process evaluations determine if a program is running in the

way it is supposed to run
Impact evaluations determines if a program creates a change
in an outcome(s)?
Did concrete encased springs decrease diarrhea rates?
What was the impact?
66% reduction in source water e coli concentration
24% reduction in household E coli concentration
25% reduction in incidence of diarrhea
Introduction Components of Program Evaluation

Impact Evaluation

Process evaluation

Process evaluations determine if a program is running in the

way it is supposed to run
Impact evaluations determines if a program creates a change
in an outcome(s)?
Did concrete encased springs decrease diarrhea rates?
What was the impact?
66% reduction in source water e coli concentration
24% reduction in household E coli concentration
25% reduction in incidence of diarrhea
Introduction Components of Program Evaluation

Impact Evaluation

Process evaluation

Process evaluations determine if a program is running in the

way it is supposed to run
Impact evaluations determines if a program creates a change
in an outcome(s)?
Did concrete encased springs decrease diarrhea rates?
What was the impact?
66% reduction in source water e coli concentration
24% reduction in household E coli concentration
25% reduction in incidence of diarrhea
Introduction Components of Program Evaluation

Impact Evaluation

Process evaluation

Process evaluations determine if a program is running in the

way it is supposed to run
Impact evaluations determines if a program creates a change
in an outcome(s)?
Did concrete encased springs decrease diarrhea rates?
What was the impact?
66% reduction in source water e coli concentration
24% reduction in household E coli concentration
25% reduction in incidence of diarrhea
Introduction Components of Program Evaluation

Impact Evaluation

Process evaluation

Process evaluations determine if a program is running in the

way it is supposed to run
Impact evaluations determines if a program creates a change
in an outcome(s)?
Did concrete encased springs decrease diarrhea rates?
What was the impact?
66% reduction in source water e coli concentration
24% reduction in household E coli concentration
25% reduction in incidence of diarrhea
Introduction Components of Program Evaluation

Impact Evaluation

Making policy from evidence

Which program should we invest in? What are the issues in

choosing the program?
Introduction Components of Program Evaluation

Cost-Effectiveness Analysis

1 Introduction
Evidence-Based Policy Making
What is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation

2 Components of Program Evaluation

5 Components of Program Evaluation
Identifying the Problem
Theory of Change
Impact Evaluation
Cost-Effectiveness Analysis
Introduction Components of Program Evaluation

Cost-Effectiveness Analysis

Cost-effectiveness diagram

Which program should we invest in? What are the issues in

choosing the program?