IPDET

Module 4:
Understanding
the Evaluation Context and
Program Theory of Change
 Introduction
• Front-End Analysis
• Identifying the Main Client and Key
Stakeholders
• Understanding the Context
• Investigating Existing Knowledge
• Constructing, Using, and Assessing a Theory
of Change

2
IPDET © 2009
 Front-End Analysis
• Client and stakeholders
• Timing
• Time management
• Nature of resources
• Relevance of social science theory
• Findings of similar evaluations
• Role of theory of change
• Existing data
3
IPDET © 2009
 Balancing Cost and Benefits:
Benefits
• Evidence-based decision making that leads to
sound decisions about:
– what to scale up or replicate
– What to continue
– what to improve
– what to decrease or terminate
• Contributions to streams of knowledge
• Building of local capacity

4
IPDET © 2009
 Balancing Cost and Benefits:
Costs
• Cost of the evaluation in relation to the
cost of the program
• Burden of the evaluation on those who
must supply information
• Reputation and political costs to the
organization, the evaluator, and the
evaluation community if the evaluation
is not done well
5
IPDET © 2009
 Dangers or Pitfalls
• Resistance to modifying the original plan
• McDonaldization
• Fixed beliefs (“truisms”)
• “Group think”
• Disproportionate weighting of the views
of the powerful

6
IPDET © 2009
 Main Client(s)
May be those:
• authorizing or requesting the evaluation
• authorizing or funding the program
• accountable to the public for the intervention
• to whom the evaluators are accountable

7
IPDET © 2009
 Stakeholders
• People or organizations other than the
client with strong interests in the
intervention and its evaluation
– May stand to gain or lose from the
outcomes
– May represent the interests of such groups
• Important to consider those who would
typically not be asked to participate
8
IPDET © 2009
 Checklist of Stakeholder
Roles
Individuals, groups, To To make To provide To For
or agencies make operational input to react interest
policy decisions evaluation only
Developers of
program
Providers of other
resources

Heads of
implementing
agencies
etc.

9
IPDET © 2009
 Involving Stakeholders
• Identify the main stakeholders
• Provide early opportunity for them to raise
issues for the evaluation to address
• Consider offering periodic updates or
briefings
• Evaluation report should indicate the
nature and extent of stakeholder
involvement
10
IPDET © 2009
 Stakeholder Analysis
• A technique to identify and assess the
importance of key people, groups, or
institutions
• Assists in anticipating the influence
stakeholders may have on
– mobilizing support or resistance to the evaluation
– use of evaluation findings and implementation of
recommendations
• Evaluator may develop strategies to involve
appropriately in the evaluation
11
IPDET © 2009
 Program Stages and Evaluation
Questions
Program Stage Examples of evaluation questions
1. Prospective What can be learned from the experiences of similar
programs?
2. Early implementation Is the program operating as planned? What issues
are surfacing?
3. Mid-implementation To what extent are there cross-site variations in how
the program is being implemented?
4. Maturity To what extent have the outcomes been achieved?
Can gains be attributed to the program? Has the
program had unanticipated positive or negative
effects?
5. Ex-poste To what extent did the program contribute to the
observed impacts?

12
IPDET © 2009
 Tapping Existing
Knowledge
• Own organization’s internal reviews and
published evaluations
• Other organizations’ websites with
evaluation publications
• Repositories, syntheses, or collections of
evaluations
• Evaluation journals
• Doctoral dissertations
• Related literature: e.g. economics, statistics
13
IPDET © 2009
 Using Knowledge for a
“Theory of Change”
• Identify stream of knowledge that explains or
predicts an event (a theory)
– example: evidence from evaluative research that
indicates students do better when parents are
involved in their studies
• Use this knowledge to develop a “theory” of
how the program will achieve intended change
– example: if parents in rural Zambian primary schools
are more involved in their children‘s studies, the
children will do better in their studies
14
IPDET © 2009
 Theory of Change
• A knowledge-based diagram of how an
intervention intends to achieve results
• T of C should:
– depict a sequence linking inputs to activities,
activities to outputs, and the results--outcomes and
impacts– expected from them.
– identify critical assumptions underlying the
intervention
– Indicate other factors in the context of the
intervention that might influence its success
15
IPDET © 2009
 Potential Influences on
Program Results
Political Environment
Policy Context (Governance, etc.)

Macro-economic
Picture Inputs  Activities  Outputs Public Attitudes

Black box

Aid Players Outcomes  Impacts Environment

16
IPDET © 2009
 Why Use a Theory of
Change
• Visually shows how the intervention is
supposed to work and flaws in logic
• Assumptions and potential influences
identified may indicate risks to the
achievement of intended results
• Provides questions for an evaluation
• Helps build a common understanding of the
intervention and expectations among
stakeholders
17
IPDET © 2009
 Review or Create
• If the intervention already has a
developed theory of change, carefully
review it
• May need to refine, rework, or
reconstruct
• If there is no theory of change, create
one

18
IPDET © 2009
 Constructing a Theory of
Change
• What knowledge underlies the
intervention? [Slide 13]
• What is the logic of the intervention?
• What are the key assumptions being
made?
• What are potential influences operating in
the context of the intervention?

19
IPDET © 2009
 How to Determine the
Logic of the Intervention
• Chain of “if-then” statements
– If X happens, then Y should follow
• Ask, is this plausible? Are the activities and
outputs likely to lead to the intended results?
• Ask, is this feasible? Are the inputs
(capabilities and resources) sufficient for the
activities?

20
IPDET © 2009
 Mapping or Drawing the
Logic
• Can and are done in many different
ways
• Usually use forms of flow-charting
• Boxes with arrows show links and
relationships

21
IPDET © 2009
 Results Chain

Area of Control Outputs Reach Area of Influence

Internal to the Direct External to the
Organization Beneficiaries Organization

Inputs Activities Outputs Outcomes Outcomes Impacts

(Resources) (Short-Term) (Long
(Intermediate)
term)

External Factors

22
IPDET © 2009
 Results Chain for Training
Program
Inputs → Activities → Outputs → Outcomes → Impacts

Resources Services Products Benefits Changes

Money Training Total number New knowledge Trainees earn
Staff Education of classes Increased skills more over five
Hours of years than those
Volunteers Counseling Changed not receiving
service attitudes
Supplies training
Number of New
• Eligible participants Trainees have
participants employment higher standard
completing opportunities
course of living than the
control group

23
IPDET © 2009
 Simple Theory of Change
Model

Training High Quality Useful Better

Evaluations Information Decisions

By training We should get

better decisions

24
IPDET © 2009
 Identifying Key Assumptions

Make explicit the implicit assertions

about how the intervention will work in
its context to produce the results

25
IPDET © 2009
 Theory of Change with
Assumptions
Report writing skills
Needs of adequate to
trainees met communicate with
government

Training High Quality Useful Better

Evaluations Information Decisions

Enough time Resources given

to learn to follow what
they learned

26
IPDET © 2009
 Theory of Change with
Assumptions and Context
Macro
economic Foundation
environment programs
Report writing skills
Needs of adequate to
Government- trainees met communicate with NGO’s
funded government programs
related
programs

Training High Quality Useful Better

Evaluations Information Decisions

Enough time Resources given

Worker
to learn to follow what
remittances
they learned
Other bilateral
micro-financed
MDM programs

27
IPDET © 2009
 Flow Chart of Classic
Logic Model
Inputs

Activities

Outputs

Outcomes

Impacts
28
IPDET © 2009
 Exercise: Unlabeled
Theory of Change
Visits by teachers to students’ homes

Sharing of views by parent and teacher

Teachers’ Parents’ knowledge Identification of

understanding of of school’s special problems that
the home culture expectations for retard student’s
students achievement (health,
emotional, etc.)

Teachers’ Parental support and Parents’ knowledge

Parental support
sympathy with encouragement with of school’s
for better
children and their child’s homework and expectations for
attendance at
view of the world school assignments students
school

Teaching in terms Conscientiousness Student attendance Student’s receipt of

comfortable and of work by students special help
understandable to
students
Improvement of
Student morale condition (health,
emotional)

Achievement in reading

29
IPDET © 2009
 Women have limited access to economic opportunities
due to lack of access to credit and other productive
resources and to social control
Worker
remittances
Government-
funded related
Project offers credit, technical
programs
assistance and group Other bilateral
information micro-financed
MDM programs

Macro economic
environment Women create
business Foundation
programs

NGO’s
Generate profits programs

Short-term improvement in Profits re-invested

household welfare

Permanent improvement in Business sustained

household welfare

Impact nutrition, Improved

health, and housing
clothing

Improved Economic
education for girls improvements

30
IPDET © 2009
 Model with Assumptions &
Influences
Influence Influence

Inputs Activities Outputs Outcomes Impact

Influence Influence

Assumptions
31
IPDET © 2009
 Theories of Change should
Address these Questions:
• Is the intervention built on a knowledge base?
• Are there gaps in the logic chain?
• Are interrelationships clear and plausible?
• Is it realistic to assume that the intervention will
result in the attainment of stated goals?
• On what critical assumptions is the intervention
based?
• What else is operating in the context of the
intervention that might affect results?

32
IPDET © 2009
 A Final Note….
“One does not discover new lands without
consenting to lose site of the shore for a long time.”
-- Unknown

Questions?

IPDET © 2009