Public Program

Evaluation
Logic Model and Program Theory
10-6-2020
What is Program Theory ?
• It is a Theory of Change (TOC).
• Conception of what must be done to bring about intended
social changes.
• Explains why a program does what it does.
• Provides a rationale as to why a program should produce its
intended results.
• Also called...program model, action theory, or logic model.
Theory of Change
• T.O.C. is a comprehensive
description and illustration of
how and why a desired
change is expected to happen
in a particular context. It is
focused in particular on
mapping out or “filling in”
what has been described as
the “missing middle” between
what a program or change
initiative does (its activities or
interventions) and how these
lead to desired goals being
achieved.
Theory of change, cont.
1. It does this by first identifying the desired long-term goals
and then works back from these to identify all the
conditions (outcomes) that must be in place (and how these
related to one another causally) for the goals to occur.
2. These are all mapped out in an Outcomes Framework. One
of which is the logic model.
Theory leads to Accuracy
A4 Explicit Program and Context
Descriptions Evaluations should document
programs and their contexts with
appropriate detail and scope for the
evaluation purposes.
What does it look Like?

Community Specific Intended

Need/Problem Intervention Outcomes
Example

Families Suffer
from Poor Nutrition
Healthier Families
nutrition related education
health problems
How Can Program Theory Be Useful?
• PROGRAM PLANNING: to articulate the connections between
existing social conditions, program elements, and desired
changes.
• PROGRAM IMPROVEMENT: to identify any gaps in logic
relating components of the program to the intended effects.
• EVALUATION: to help formulate evaluation questions or focus
the evaluation.
Accurate Reasoning
A7 Explicit Evaluation Reasoning
Evaluation reasoning leading from
information and analyses to findings,
interpretations, conclusions, and judgments
should be clearly and completely
documented.
Assumptions & IF, then
relationships
•  Underlying a logic model is a series of ‘if-then’ relationships
that express the program’s theory of change
• IF, then -- IF, then – IF, then --- IF, then– IF, then

If
Then
Background: Development of
the Logic Model
• It is a tool for communicating program theory
*Articulated program theory
*Implicit program theory
•  Determine the purpose of a logic model
*Who will use it? For what?
•  Involve others
 *Set boundaries for logic model
• Level of specificity
 *Understand situation
 *Explore research, knowledge base, what others are
doing/have done
Background: Logic Models Provide a
Foundation for Evaluation
* Evaluability assessment
•  Provides the program description that guides our
evaluation process
•  Helps us match evaluation to the program
•  Helps us know what and when to measure
• Are you interested in process and/or outcomes?
•  Helps us focus on key, important information
* Prioritize: where will we spend our limited evaluation
resources?
3* ASK: What do we really need to know??
Why Develop a LM?
• Generates a clear and shared understanding of how a
program works
• Supports program planning and improvement
• Supports measures for reliability and validity
• Serves as a basis for evaluation
A6 Sound Designs and
Analyses
Evaluations should employ technically
adequate designs and analyses that are
appropriate for the evaluation purposes.
Accuracy Standards A2 and A3
• Valid Information • Reliable
• Evaluation Information Evaluati
information should on procedures should
serve the intended yield sufficiently
purposes and dependable and
support valid consistent
interpretations. information for the
intended uses.
What is a Logic Model
• After the first step of laying out the long-term goals and a
simple change framework, comes a more detailed stage of the
mapping process.
• Building upon the initial framework, we continue to map
backwards until we have a framework that tells the story we
think is appropriate for the purposes of planning.
• Sometimes, this will require much more detail because
stakeholders want to identify the “root” causes of the problem
they hope to resolve. In other cases, the map will illustrate
three or four levels of change, which display a reasonable set
of early and intermediate steps toward the long term goal.
Visual Depiction of Program
and T.O.C.
• Communicated how a program works by
depicting the intended relationships
among program components:
1. Inputs or resources
2. Activities
3. Outputs
4. Outcomes (ST, MT and LT)
Logic Model Flow: The Visual

ST MT LT
Inputs Activities Outputs
Outcomes Outcomes Outcomes
Example: AmeriCorps After
school tutoring program

Community
Students
Community volunteers
Community reading at
AmeriCorps volunteers placed in
Program volunteers proficiency
Funding members trained schools and
Staff (ST and
(# of ) ( and ready tutoring
outcome) advancing
to serve) students
(LT)
(MT)
Logic Model: Purpose
• Design and planning
• To clearly communicate the central goal of an initiative, why it is important,
and what will result from it.
• To select specific and measurable objectives expected to lead to the desired
result.
• To identify activities and resources needed to meet selected objectives.
• Implementation
• To identify and collect data need to effectively monitor initiative activities
and improve programming.
• To create a common reference point for everyone involved in the initiative.
• Evaluation
• To help determine if activities are being implemented as planned.
• To help determine if the initiative is making progress.
• And, ultimately, to determine if the initiative has led to the desired result.
Logic Model: Components
• Goal(s)
• Objective(s)
• Activities
• Input/Resources
• Outputs
• Outcome(s)/Impact(s)
• Indicator(s)
Activities
• are the processes, tools, events, and actions that are used to
bring about a programs changes or intended results.
Examples:

• Workshops on healthy food options

• Food preparation counseling
• Referrals to food programs and resources.

Source: WK Kellogg Foundation Evaluation Handbook (2004),

adapted.
Outputs
• are the direct products of a programs activities s and may
include types, levels and targets of services to be delivered by
programs. These are counts.
Examples:
• # of individuals attending workshops
• # of individuals receiving services
• # of individuals receiving referrals

Source: WK Kellogg Foundation Evaluation Handbook (2004),

adapted.
Outcomes
• are the expected changes in the target population served
that result from a program’s activities and fall along a
continuum, ranging from short to long term results:
Examples:
• Short –term: change in knowledge, skills and /or attitudes
(e.g., knowledge of health foods) 1-3 years
• Mid-term: changes in behavior or action (e.g .,adoption of
health food practices) 4-10 years, depending or program
• Long term: Change sin condition or status in life (e.g, food
security) Can really be 5, 10, 20 years depending on program
and intervention
Source: WK Kellogg Foundation Evaluation Handbook (2004),
adapted.
Connecting TOC to the Logic
Model: Backwards Mapping
1. After the first step of laying out the long-term goals and a
simple change framework, comes a more detailed stage of
the mapping process.
2. Building upon the initial framework, we continue to map
backwards until we have a framework that tells the story we
think is appropriate for the purposes of planning.
3. Sometimes, this will require much more detail because
stakeholders want to identify the “root” causes of the
problem they hope to resolve. In other cases, the map will
illustrate three or four levels of change, which display a
reasonable set of early and intermediate steps toward the
long term goal.
Backward Mapping
• Rossi et al (2004) tell us that if you cannot
map backward from the outcome to the
TOC then the logic model may have failed.

• Is this the death know for the program?

NO! It allows for mid course corrections.
What accuracy measure do we
meet?
A1 Justified Conclusions and
Decisions Evaluation conclusions and
decisions should be explicitly justified in the
cultures and contexts where they have
consequences.
LM: Goals vs Objectives
• Goal: A broad statement of intended outcomes for a
program, line of business or an organization.

• Goals are typically intended to guide the formation of (more specific)

objectives that can be linked to goals.
(McDavid and Howthrorn, 2006)

• Example
• To decrease the dropout rate of at-risk students in grades 9-12 at
Washington High School
SMART GOALS
• Specific

• Measurable

• Attainable

• Result-oriented and relevant

• Time-bound
Moving from SMART Goal to
specific objectives?
• Is your objective Specific?

• If you have a specific objective, you should be able to

answer the following six questions:
• What: What do you want to accomplish?
• Why: What are the reasons, purpose or benefits of accomplishing
the objective?
• Who: Who is involved? Who are the stakeholders?
• Where: Where is it going to happen?
• Which: Which attributes are important? (Requirements and
constraints; risk and protective factors).
Measurable?
• Is your objective measurable?

• If your objectives are specific and measurable, you

should be able to establish concrete criteria for
measuring progress toward the attainment of each.
• How will you know when it is accomplished?
• What indicators will you look for to measure progress and
success?
• Are data to measure your progress and success readily available?
Or do you need to develop new measures, e.g., surveys, focus
groups, etc…?
Attainable?
• Is your objective attainable?

• Given resources available to you, the objective must be

realistic and attainable.
• Do you have sufficient financial capacity to accomplish your
objective?
• Do you have personnel with sufficient abilities and skills to
accomplish your objective (including evaluation)?
• Do you have sufficient time to accomplish your objective?
• What other types of resources do you need to attain your
objective (i.e., technology, space, equipment, etc…)?
Time-bound?
• Is your objective time-bound?

• Objectives should have starting points, ending points,

and fixed durations.
• When will you achieve this objective?
• When will you undertake activities to achieve our objective?
• When can you expect to see some short-term outcomes?
Results oriented? Relevant?
• Is your objective result-oriented and relevant?

• Ultimately, your objectives should help you achieve your

major project goal(s).
• Is it worthwhile?
• Is this the right time?
• Does this align with other efforts/needs?
Applying Concepts to
Evaluation
• It involves looking at your program and asking
• Are we doing what we said we were going to do?
• Is what we’re doing working?
• How do we know it’s working?
• Did we accomplish what we set out to accomplish?

• It is part of what you put in place beforehand to help you

run your programs.
• It should be about looking ahead not backwards.

• It is part of the continuum of planning (rolling evaluation).

Finally, Communicate!
A8 Communication and
Reporting Evaluation communications
should have adequate scope and guard
against misconceptions, biases, distortions,
and errors.