What is Program Evaluation 1

What is program evaluation?

A beginners guide
Last revised 3/2/2018

Produced by
Gene Shackman, Ph.D.
The Global Social Change Research Project
Free Resources for Methods in Program Evaluation
https://ssrn.com/author=1754255
What is Program Evaluation 2

Table of Contents
Introduction..................................................................................................................................................................................................3
Planning...................................................................................................................................................................................................5
Logic Model.................................................................................................................................................................................................7
Methods......................................................................................................................................................................................................11
Surveys..................................................................................................................................................................................................12
Administrative data...............................................................................................................................................................................16
Focus groups.........................................................................................................................................................................................17
Key informant interviews......................................................................................................................................................................18
Observations.........................................................................................................................................................................................19
Qualitative Research methods...............................................................................................................................................................20
Methods: Additional Resources.................................................................................................................................................................21
Cause.........................................................................................................................................................................................................22
Successful evaluations...............................................................................................................................................................................31
What else does an evaluator need to know................................................................................................................................................33
Putting it all together.................................................................................................................................................................................39
Additional Resources.................................................................................................................................................................................40
What is Program Evaluation 3

Introduction

In this guide, we focus on one particular kind of evaluation: program evaluation.

Basically, when beginning an evaluation, program people will often want the answer to this question:

● Does the program work? And how can it be improved?

However, there are many equally important questions

● Is the program worthwhile?

● Are there alternatives that would be better?
● Are there unintended consequences?
● Are the program goals appropriate and useful?

Evaluations, and those who request them, may often benefit from a consideration of all of these questions. This guide, however,
focuses on the first of these issues: how program evaluation can contribute to improving program services.
What is Program Evaluation 4

How does program evaluation answer questions about whether a program works, or how to improve it.

Basically, program evaluations systematically collect and analyze data about program activities and outcomes.

The purpose of this guide is to briefly describe the methods used in the systematic collection of data, and a little about the use of this
data.

Additional Resources

“Program evaluation answers questions like: To what extent does the program achieve its goals? How can it be improved? Should it
continue? Are the results worth what the program costs? Program evaluators gather and analyze data about what programs are doing
and accomplishing to answer these kinds of questions.”
American Evaluation Association blog, January 2014. http://www.eval.org/p/bl/et/blogid=2&blogaid=4

“program evaluation involves the systematic collection of information that assists stakeholders to better understand a program,
improve its effectiveness, and make decisions about future program planning.”
Public Health Program Evaluation: What is it? Last Updated: December 15, 2017
https://phc.amedd.army.mil/TOPICS/HEALTHSURV/PHAPE/Pages/WhatisProgramEvaluation.aspx
What is Program Evaluation 5

Planning

The first step is planning the evaluation. Evaluations should follow a systematic and mutually agreed on plan. Plans will typically
include the following:

● Determining the purpose of the evaluation (eg accountability, improvement, learning):

● Understanding the program or intervention to be evaluated.

Purpose of the evaluation and understanding the program will determine the next three parts of the plan:

● What is the evaluation question, what is the evaluation to find out (see next page).

● How will the evaluation answer the question: What methods will be used.

● Resources needed: e.g., personnel, resources, skills, time

Finally, the plan should include:

● How to engage the stakeholders. Helps determine the questions and that the results will be used.

● Making the results useful, how will the results be reported so that they can be used by the organization to make improvements.

Additional resources about planning evaluations :

Better Evaluation. Manage Evaluation. http://www.betterevaluation.org/en/plan/manage_evaluation

The Program Manager's Guide to Evaluation, Second Edition https://www.acf.hhs.gov/opre/resource/the-program-managers-guide-to-

evaluation-second-edition from 2010

Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).
https://www.cdc.gov/eval/framework/index.htm
What is Program Evaluation 6

A key part of evaluation is to determine the question.

“assessment of the operation and/or outcomes of a program or policy”

Evaluations can generally answer two types of questions:

1. Outcome evaluation: What is the outcome of the program? Did the program have any impact, was there any improvement in
people's lives?

2. Process evaluation: How did the program get to that outcome? Did the program have some set of procedures? Were these
procedures followed, were the procedures reasonable, was there a better way to get to the outcomes?

Additional resources:

Approaching An Evaluation-- Ten Issues to Consider

Brad Rose Consulting, Inc.
https://bradroseconsulting.com/approaching-an-evaluation/
What is Program Evaluation 7

Logic Model

One way to figure out the questions is for the evaluator and program people to develop a very good description of:
● What the program/intervention is doing
● What the outcomes should be,
● How the program will get there, and
● Why the program leads to the outcome.

This description helps to identify how the program should lead to the outcome, why the program activities should lead to the
outcomes, and where to evaluate the program to check whether it does.

This method is called a program theory.

“A program theory explains how and why a program is supposed to
work. ... It provides a logical and reasonable description of why the things you do – your program activities – should lead to the
intended results or benefits.”

From Program Evaluation Tip Sheets from Wilder Research, Issue 4, October 2005 - Program Theory. https://dps.mn.gov/divisions/ojp/forms-
documents/Documents/Wilder_Program_Evaluation_4.pdf
What is Program Evaluation 8

A useful tool to help work with the program theory is a logic model, which visually shows the program theory, how all the
program goals, activities, and expected outcomes link together.

Logic model example:

from
The Logic Model for Program Planning and Evaluation
Paul F. McCawley, Associate Director, University of Idaho Extension. 2001.
http://www.cals.uidaho.edu/edcomm/pdf/CIS/CIS1097.pdf
What is Program Evaluation 9

Use the program theory or logic model to come up with evaluation questions, for example:
● Does the program have a positive outcome?

● Are people satisfied?

● How could the program be improved?

● How well is the program working?

● Is the program working the way it was intended to work?

Additional Resources

Developing Evaluation Questions

Central Comprehensive Center at the University of Oklahoma
http://c3ta.org/knowledgebases/knowledgebases/resources/developing-evaluation-questions-820.html

Developing Process Evaluation Questions. At the National Center for Chronic Disease Prevention and Health Promotion. Healthy
Youth. Program Evaluation Resources
https://www.cdc.gov/healthyyouth/evaluation/index.htm
What is Program Evaluation 10

However, there are limits to program theory and logic models:

● Models are linear, programs are complex, interactive

● Models are static, programs may change over time.
● Models may not take unexpected consequences into account
● Models may not account for conflict, power, control issues
● Theory or model assumes the model is correct.
● Model may under-emphasize processes or external influences.

Use program theory and logic models, but be flexible, and open to change and feedback. Constantly review, re-assess, and revise
them, as necessary.

Additional Resources about logic models.

Jonathan A. Morell, PhD. Workshop: Logic Models: Uses, Limitations, Links to Methodology and Data. 2012. http://jamorell.com

Program logic - an introduction. from Audience Dialogue

http://www.audiencedialogue.net/proglog.html

Dana Petersen, Ph.D., Erin F Taylor, Ph.D., and Deborah Peikes, Ph.D. The Logic Model: The Foundation to Implement, Study, and
Refine Patient-Centered Medical Home Models. March 2013. AHRQ Publication No. 13-0029-EF .
https://pcmh.ahrq.gov/page/logic-model-foundation-implement-study-and-refine-patient-centered-medical-home-models
What is Program Evaluation 11

Methods

There are many methods, each with their own uses, advantages and difficulties. Methods include:

Surveys
Analysis of Administrative Data
Key Informant Interviews
Observation
Focus Groups

Evaluations could use any, not necessarily all, of these methods, depending on the question and goal of the evaluation.

Additional Resources

Overview of Basic Methods to Collect Information

Carter McNamara, MBA, PhD, Authenticity Consulting, LLC
https://managementhelp.org/businessresearch/methods.htm

Determine Collection Method

Evaluation Toolkit. Pell, Institute.
http://toolkit.pellinstitute.org/evaluation-guide/collect-data/determine-collection-method/
What is Program Evaluation 12

Surveys
Surveys are a set of questions that are asked of everyone in the same way.

Surveys can answer question about how many and how often. For example:

● How many clients are satisfied with services?

● How often do people have difficulties using the services?

Typical questions might be like this:

How satisfied are you with the program?

very satisfied neither dissatisfied very
satisfied dissatisfied

How did you hear about the program? Check all that apply.

□ Radio
□ TV
□ friends
□ other _____________________

Surveys might be used to describe the entire client population, if respondents were chosen randomly or systematically (see next page)
and if the sample is sufficiently large.
What is Program Evaluation 13

Survey Administration: Generally, the goal is to administer a survey so the results can describe a larger population. That
means the survey should be administered to people using some clearly defined method of selection. For example:

● Random selection – generate a list of random numbers, assign each person a random number, sort the people by the random
number and take the people listed first. They were put on top of the list randomly.

● Systematic selection – a typical method is to start with the 5th person and then select every 7th person after that. The numbers,
the 5th and the 7th are also chosen randomly.

● Randomly select locations to be in the sample, and then survey everyone in that location.

Random or systematic selection means that the group of people you select are more likely to be similar to your clients, in general. You
aren't excluding any particular groups, or including only certain groups. You are avoiding bias, in sampling terms.

If you do use random or systematic selection, then it is more likely that the sample represents the population and you can use the
results of your survey to make conclusions about your clients.

If you don't use random or systematic selection, you can NOT use the results of your survey to make conclusions about your client
population. That is, you cannot generalize from your study to your client population. You can only say “The people who took this
survey said ...”
What is Program Evaluation 14

Survey Administration:
Surveys can be cross sectional.

Cross sectional means the surveys are given to many people in the same time frame, for example in the same month or year. This is a
snapshot of the population during a single time period.

Cross sectional surveys can be used to compare different groups. For example, men said xyz while women said abc. National opinion
polls are typically cross sectional surveys.

It is difficult for cross sectional surveys to establish cause, because they often cannot establish event order. Surveys can ask people
about past events, but usually the further back or less important past events are, the less accurate the memory (recall bias).

Surveys can also be longitudinal. These kinds of surveys can follow the same people over time. Longitudinal surveys can study how
people change over time and the order of events associated with the changes.

Longitudinal surveys can help to reduce recall bias. Also, because these surveys follow people over time, they can show whether
changes came before or after events, and so can help in establishing causality.

Potential problems with longitudinal surveys have to do with how people leave the survey group. People may move, just want to quit,
or leave the survey for many other reasons. If the pattern of people leaving is not random, then the remaining sample may be bias.
That is, if those who leave are different from everyone else, then those who stay may no longer be a random sample, which could
affect the validity of the results.
What is Program Evaluation 15

Additional Resources about surveys

What researchers mean by... cross-sectional vs. longitudinal studies. Institute for Work and Health. At Work, Issue 81, Summer 2015.
https://www.iwh.on.ca/wrmb/cross-sectional-vs-longitudinal-studies

Introduction to Longitudinal studies. UCL Institute of Education

https://learning.closer.ac.uk/introduction/

Introduction to study designs - cross-sectional studies

https://www.healthknowledge.org.uk/e-learning/epidemiology/practitioners/introduction-study-design-css

What is a Survey, by Fritz Scheuren. http://www.whatisasurvey.info/

Evaluating Survey Quality in Today's Complex Environment

American Association for Public Opinion Researcher
http://www.aapor.org/Education-Resources/Reports/Evaluating-Survey-Quality.aspx

Sampling. Pew Research Center. http://www.pewresearch.org/methodology/u-s-survey-research/sampling/

Social Science Research: Principles, Methods, and Practices

Anol Bhattacherjee, University of South Florida
Chapter 8 is on Sampling. Chapter 9 is on Surveys
http://scholarcommons.usf.edu/oa_textbooks/3/

Recall Period in Consumer Expenditure Surveys Program. Norman M. Bradburn. NORC at the University of Chicago. December
2010 Methods Workshop. BLS Consumer Expenditure Survey, Gemini Project Materials.
https://www.bls.gov/cex/geminimaterials.htm
What is Program Evaluation 16

Administrative data
Analysis of administrative data is just using statistical analysis on program data that is already collected.

Administrative data has advantages:

● No new data collection is required

● Many databases are relatively large
● Data may be available electronically

and disadvantages:

● Data were gathered for another purpose, so may not have necessary variables.
● In all administrative data sets, some fields are likely to be more accurate than others.

Additional Resources

Use of administrative data. Statistics Canada

http://www.statcan.gc.ca/pub/12-539-x/2009001/administrative-administratives-eng.htm

Data collection: Types of data collection – Administrative Data.

Statistics Canada.
http://www.statcan.gc.ca/edu/power-pouvoir/ch2/types/5214777-eng.htm
What is Program Evaluation 17

Focus groups
Focus groups are structured discussions among small groups of people.

Generally, a facilitator leads a group of 8-10 people in a discussion about selected topics with planned questions, while allowing for
interesting, new or unplanned follow up questions.

Typical focus group questions are like these:

● What is your overall impression of the program?

● What are the things you like or dislike about the program?
● What have you gained in this program?
● If you have not noticed any changes in yourself, what do you think are the reasons?

Additional Resources about focus groups

Basics of Conducting Focus Groups

Carter McNamara, MBA, PhD, Authenticity Consulting, LLC
https://managementhelp.org/businessresearch/focus-groups.htm

Community Toolbox: Conducting Focus Groups.

http://ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/conduct-focus-groups/main
What is Program Evaluation 18

Key informant interviews

Key informant interviews are qualitative, in-depth interviews of 15 to 35 people selected for their first-hand knowledge about a topic
of interest.

Tips for Conducting Key Informant Interviews

SAMHSA https://www.samhsa.gov/capt/tools-learning-resources/conducting-key-informant-interviews

Although key informant interviews are more informal than other forms of data collection, they still require a structure to be effective.
Your respondent is more likely to take you seriously (and provide better information) if you are prepared and the conversation has
direction.

Begin by introducing your project and purpose. Start with an easy question. Ask your most important questions first

Additional Resources

UCLA Center for Health Policy Research

Section 4: Key Informant Interviews
http://healthpolicy.ucla.edu/programs/health-data/trainings/Documents/tw_cba23.pdf

Key Informant Interviews

University of Illinois Extension
http://ppa.aces.uiuc.edu/KeyInform.htm
What is Program Evaluation 19

Observations
Observations are methods that yield a systematic description of events or behaviors in the social setting chosen for study.

Observation methods can be highly structured, for example:

Systematic Social Observation - a field research method in which teams of researchers observe the object of study in its natural
setting. Researchers record events as they see and hear them and do not rely upon others to describe or interpret events. The
researchers follow well-specified procedures that can be duplicated.

Above from: Systematic Observation of Public Police: Applying Field Research Methods to Policy Issues. Stephen D. Mastrofski, Roger B. Parks, Albert J.
Reiss, Jr., Robert E. Worden, Christina DeJong, Jeffrey B. Snipes, William Terrill. National Institute of Justice, December 1998.
https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=172859

Observations can also be unstructured, for example, participant observation, or taking an active part in group activities.

The premise underlying participant observation “is that the researcher becomes a more effective observer by taking an active role in
the performance of regular activities. In other words, knowledge gained through doing is of a higher quality than what is obtained only
through observation. In many cases, involvement with ordinary chores will not only enhance the researcher's understanding of the
processes, techniques, and words associated with these activities, but will also result in better rapport with informants.”

Above quote from: Documenting Maritime Folklife: An Introductory Guide, Part 2: How to Document. Participant Observation. American Folklife Center.
Library of Congress. http://www.loc.gov/folklife/maritime/twopo.html
What is Program Evaluation 20

Qualitative Research methods

Focus groups, interviews and observation are qualitative research methods, that is, methods that are less likely to rely on statistical
analysis.

Advantages

● Can help explain the “how” and “why”, through gathering greater detail, and more in depth responses from participants.
● Because these methods are less structured and more flexible, can discover information that might be unexpected or not planned
for.
● Can be used when little is known about the topic/situation. Surveys, for example, are hard to use when don’t know enough to
know what questions to ask.
● Can help to generate further questions, suggestions, recommendations.

Disadvantages

● The evaluator's subjective views can introduce error. Difficult to validate.

● The focus of the evaluator is only on what is observed at one time in one place, and samples are usually small and non-
representative, so usually cannot generalize to the client population.
● Information from observations/ interviews/ groups can be time consuming and difficult to interpret.
● Focus groups could be dominated by one individual and their point of view.
● Qualitative data is less useful in establishing causal relationships.

Additional Resource

Community Toolbox. Section 15. Qualitative Methods to Assess Community Issues

https://ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/qualitative-methods/main
What is Program Evaluation 21

Methods: Additional Resources

Simply Psychology, Saul McLeod

Chapter on Research Methods
https://simplypsychology.org/research-methods.html

e-Source http://www.esourceresearch.org/ From the US NIH. This site says "Inside you will find 20 interactive chapters with
authoritative answers to methodological questions on behavioral and social science research." Basically, how they say research should
be done.

Anol Bhattacherjee Social Science Research: Principles, Methods, and Practices. 2012 http://scholarcommons.usf.edu/oa_textbooks/3/
What is Program Evaluation 22

Cause

Did the program have an effect?

The ultimate goal of a program is to improve people's lives. How do you know whether it did?

One commonly used way to find out whether the program improved people's lives is to ask whether the program caused the outcome.
If the program caused the outcome, then one could argue that the program improved people's lives.

On the other hand, if the program did not cause the outcome, then one would argue that, since the program did not cause the outcome
then the program did not improve people's lives.

How to figure this out?

Determining whether a program caused the outcome is one of the most difficult problems in evaluation, and not everyone agrees on
how to do it. Some say that randomized experiments are the best way to establish causality. Others advocate in-depth case studies as
best. The approach you take depends on how the evaluation will be used, who it is for, what the evaluation users will accept as
credible evidence of causality, what resources are available for the evaluation, and how important it is to establish causality with
considerable confidence. (This paragraph suggested by Michael Quinn Patton.).
What is Program Evaluation 23

There are three approaches frequently used to establishing whether a program or intervention causes an outcome.

• Experimental or quasi experimental design – for example using comparison groups, comparing people in the program to
people not in the program

• multiple evaluation methods – comparing results from several evaluations, each using different methods

• in depth case studies of programs and outcomes – showing that the links between what program participants experience and
the outcomes attained are reasonable, empirically validated, and based on multiple sources and data. The linkages between
program and outcomes are direct and observable. No alternative possible causes offer a better explanation. (See the in depth
case study section later.)

The particular method that is used should reflect a careful discussion and understanding of the pros and cons of each method, and
agreement among all parties involved.
What is Program Evaluation 24

Comparisons and cause:

Comparison groups and random assignment

The idea is this:

Randomly assign people to either be in the program (the 'treatment' group) or to be not in the program (the 'control' group).

Since people in the 'treatment group' and ‘control group’ were randomly assigned, then before the program the two groups of people
should be pretty much the same.

Measure the treatment group after they have been on the program and compare them to people in the control group.

After the program, if the 'treatment' group people are better off than are the control group people, then the difference should be from
being in the program, and so it is reasonable to argue that the program caused that outcome.

Additional Resources:

When to Use a Randomized Controlled Trial and When Not to. Bernadette Wright. http://meaningfulevidence.com/publications

Why randomize. Institution for Social and Policy Studies. http://isps.yale.edu/node/16697

Randomized Control Trial. Angela Ambroz, and others. Better Evaluation. http://www.betterevaluation.org/en/plan/approach/rct
What is Program Evaluation 25

Advantages and disadvantages of random assignment to treatment and comparison groups.

Advantages:

● Results provide clearest demonstration of whether a program causes an outcome.

● Provides results that are easiest to explain.

Disadvantages:

● Often not practical to do. Can't randomly assign people to program or not program, and may be unethical to randomly assign
someone to no treatment.
● Randomly assigning people to be in the program is not how programs really work, so results of the evaluation may not apply to
the program as it really exists.
● Can't be applied to causes that operate over the long term or to programs that are very complex.
● Can tell whether a program caused outcome, but doesn't give much in depth information about why or how.
● People in treatment group know they are getting treatment so outcome may be due to knowledge, not to treatment.

This is a summary of points from:

(Munck and Jay Verkuilen) “Research Designs,”
http://www-bcf.usc.edu/~munck/research/methods.html

Last point from: A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research. Michael Scriven. Journal of MultiDisciplinary
Evaluation, Volume 5, Number 9. http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/160
What is Program Evaluation 26

Comparison groups and non random assignment When random assignment is not possible, quasi-experimental design can be used.
There are many reasons why random assignment can’t be used. In some cases, random assignment may be unethical. If “no treatment”
is unsafe, then cannot randomly assign people to treatment or no treatment. Random assignment may not be practical, for example, if
an existing program is open to everyone, so cannot control who participates and who does not.

There are many types of quasi-experimental design. These are a few.

● Comparing people already on the program to those who are not on the program. One example is to observe (O) people before
they join the program or there is an intervention (X), then observe both groups after :

Pretest-posttest design
-Intervention group O1 X O2
-Comparison group O1 O2

● Measuring the client many times before they join the program (or before a new intervention) and many times afterward, them
compare before to after. One example is:

Time series design

-Intervention group O1 O2 X O3 O4

● Combination of the two above

Time series design

-Intervention group O1 O2 X O3 O4
-Control group O1 O2 O3 O4
What is Program Evaluation 27

A major challenge to non random assignment approaches is that people on the program may start off being very different from the
people not on the program.

That is, only some people choose to be on the program. Something made these people different and it may be the something which
caused the better outcome, not the program.

One way to deal with this is to collect as much information as possible on characteristics of the people and program that relate to the
program outcome (what the program is supposed to do), and use this information in statistical analysis to “control” for the differences
between people on the program vs people not on the program.

The problem is that there may be differences, some critical, that are not observed, and for which the evaluator has no data.

Additional Resources:

AllPsych On Line. By Dr. Christopher L. Heffner

Section 5.3 Quasi-Experimental Design
http://allpsych.com/researchmethods/quasiexperimentaldesign.html

Study Design 101

http://himmelfarb.gwu.edu/tutorials/studydesign101/

Diagrams on previous page from:

Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach. Stage 4, Planning Evaluation. National
Network of Libraries of Medicine
https://collections.nlm.nih.gov/catalog/nlm:nlmuid-100961143-bk

Quasi-Experimental Evaluation, Kristin Anderson Moore, Ph.D. 2008

http://www.childtrends.org/?publications=quasi-experimental-evaluations
What is Program Evaluation 28

Multiple evaluation methods could support the idea that the program causes the outcome if different sources agree.
For example, collect information from:

● Program participants
● Program staff
● Community members
● Subject experts
● Published research and reports

Collect data through many methods, for example:

● Surveys
● Interviews
● Observations
● Program administrative data

If data from different sources don't agree, it doesn't necessarily mean the results from any of the sources are not valid. However, the
more agreement there is from different sources, the more confident you can be about your conclusions.

Additional Resources:

Encyclopedia of Social Science Research Methods: Triangulation

Contributors: Alan Bryman, 2004.
https://edge.sagepub.com/system/files/Triangulation.pdf

Triangulation In Social Work Reserach: The Theory and Examples of its Practical Applications. Mike Weyers, Herman Strydom, Arnel
Huisamen. Social Work/Maatskaplike Werk 2008:44(2) http://socialwork.journals.ac.za/pub/article/view/251
What is Program Evaluation 29

An in depth case study can be used to demonstrate the connection between the intervention and the outcome.
An in-depth case study documents in detail what a group of participants experienced in a program and any ways in which they have
changed so that the evaluator and users of the evaluation can make a judgment about the likelihood that the program led to the
observed changes. For example, a group of chronic alcoholics go through a residential chemical dependency program. Their
participation is fully documented. They return home maintaining their sobriety. They attribute their sobriety to the program as do their
families, friends, and program staff. These multiple sources agree on the documented causal chain. The links between what they
experienced and the outcomes attained are reasonable, empirically validated, and based on multiple sources and data. The linkages
between program and outcomes are direct and observable. No alternative possible causes offer a better explanation. The
preponderance of evidence points to a conclusion about causality. Establishing causality involves both data and reasoning about the
findings. (This paragraph contributed by Michael Quinn Patton.)

Cause - Summary:

Random assignment is often seen as a very clear way to show whether a program causes an outcome. However, random assignment is
often not practical or reasonable.

Other methods such as non random assignment, multiple evaluation methods, or in depth case studies may be more practical and can
be used to give reasonable arguments about whether a program caused an outcome.

However, these methods are less certain in establishing that the program is the cause of the outcome. There may be other things going
on that are unknown and these other things might really be the cause of the outcome. It is more difficult for these other methods to rule
out other possible causes, although the other methods can, again, establish reasonable likelihood. If a cause cannot be established, the
evaluation can be used to describe what happened.

For example, the evaluation could say, “After the program, people were better off.” This doesn't necessarily mean it was the program
that made the people better off, but the program may be one reasonable cause. Gathering more evidence, from multiple methods,
driven by a very clear understanding of the program, can help determine the most reasonable explanation of causes of the outcomes.
What is Program Evaluation 30

Additional Resources about design:

Steps in Program Evaluation

Gathering Credible Evidence
https://www.cdc.gov/eval/steps/gaterhingcredibleevidence.pdf

Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective Interventions
GAO-10-30 November 23, 2009
http://www.gao.gov/products/GAO-10-30

What Constitutes Credible Evidence in Evaluation and Applied Research? Claremont Graduate University Stauffer Symposium
https://research.cgu.edu/dbos-events-and-conferences/home/what-constitutes-credible-evidence-in-evaluation-and-applied-research

A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research. Michael Scriven. 2008. Journal of
MultiDisciplinary Evaluation, Volume 5, Number 9.
http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/160
What is Program Evaluation 31

Successful evaluations
One way to increase likelihood of successful evaluation is by using a collaborative approach.

Involve many stakeholders in decisions about the evaluation: how it is going to be conducted, and how the results will be interpreted
and used.

Involving stakeholders may:

● reduce suspicion
● increase commitment
● broaden knowledge of evaluation team
● increase the possibility that results will be used.

Additional Resources:

Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. Step 1: Engage Stakeholders.
https://www.cdc.gov/eval/guide/step1/

Better Evaluation. Understand and engage stakeholders

http://www.betterevaluation.org/en/plan/manage/identify_engage_users
What is Program Evaluation 32

Another way is to plan the evaluation from the start of the program or intervention.

“Having a plan helps ensure that future evaluations are feasible and instructive.” (Martin, NIJ Journal, September 2015)

So for example, knowing the questions to be asked can help the program be designed to gather the information needed.

Having a plan in writing also makes the evaluation transparent.

Additional Resources:

Plan for Program Evaluation from the Start

by Alison Brooks Martin. NIJ Journal, #275. September 2015
https://www.nij.gov/journals/275/pages/plan-for-program-evaluation.aspx
What is Program Evaluation 33

What else does an evaluator need to know.

• Evaluation Standards:

Joint Committee on Standards for Educational Evaluation standards (JCSEE)

http://www.jcsee.org/program-evaluation-standards-statements
standards about utility, feasibility, fairness and legality, accuracy, and accountability.

Guiding Principals for Evaluators, American Evaluation Association http://www.eval.org/p/cm/ld/fid=51

principals about technical methods, competence, integrity/honesty, respect for people, responsibility for general
welfare.

DAC Quality Standards for Development Evaluation. 2010.

http://www.oecd.org/dac/evaluation/publicationsanddocuments.htm

United National Evaluation Group, Norms and Standards for Evaluation. 2016.
http://www.uneval.org/document/detail/1914

Canadian Evaluation Society and CDC adopted the JCSEE standards. 2012
https://evaluationcanada.ca/program-evaluation-standards
https://www.cdc.gov/eval/standards/index.htm
What is Program Evaluation 34

• Learning evaluation methods

Links to free online classes

https://sites.google.com/site/gsocialchange/learning

Examples:

Northwest Center for Public Health Practice http://www.nwcphp.org/training

Action Research and Action Learning http://www.aral.com.au/

EvalPartners e-Learning programme in Development Evaluation http://elearning.evalpartners.org/elearning

The Global Health Learning Center section on monitoring and evaluation

http://www.globalhealthlearning.org/program/monitoring-and-evaluation

Unicef http://www.ceecis.org/remf/Service3/unicef_eng/index.html

Edx current and archived classes, statistics https://www.edx.org/courses?search_query=statistics

What is Program Evaluation 35

• Statistics

Online statistics books

https://sites.google.com/site/gsocialchange/statbooks

Examples:

CADDIS Volume 4: Data Analysis

https://www.epa.gov/caddis-vol4

Electronic Textbook http://www.statsoft.com/textbook/

Gustman, Burt. Stat Primer. Last revised 2016 http://www.sjsu.edu/faculty/gerstman/StatPrimer/

Lane, David. HyperStat http://davidmlane.com/hyperstat/index.html

Stark, Philip B. SticiGui: Statistics Tools for Internet and Classroom Instruction
Last modified 2016.
http://www.stat.berkeley.edu/~stark/SticiGui/

Stockburger, David. Introductory Statistics. 2016 http://dwstockburger.com/Introbook/sbk.htm

UCLA. Probability and statistics EBook http://wiki.stat.ucla.edu/socr/index.php/EBook

What is Program Evaluation 36

• Journals, keeping up with methods, research

Online journals about evaluation methods

https://sites.google.com/site/gsocialchange/journals

Examples:

African Evaluation Journal http://www.aejonline.org/index.php/aej/index

Journal of Methods and Measurement in the Social Sciences (JMM)

https://journals.uair.arizona.edu/index.php/jmmss/index

Journal of Official Statistics http://www.degruyter.com/view/j/jos

Journal of MultiDisciplinary Evaluation http://journals.sfu.ca/jmde/index.php/jmde_1/index

Methods, data, analyses http://mda.gesis.org/index.php/mda

Qualitative Social Research http://www.qualitative-research.net/index.php/fqs/index

What is Program Evaluation 37

• Presenting statistical data

Links to sites about presenting data https://sites.google.com/site/gsocialchange/presenting

Examples:

Making Data Meaningful

http://www.unece.org/stats/documents/writing/

Gallery of Data Visualization, best and worst

http://www.datavis.ca/gallery/

Rougier NP, Droettboom M, Bourne PE (2014) Ten Simple Rules for Better Figures. PLoS Comput Biol
http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003833

Presenting Statistical Data

http://www.qgso.qld.gov.au/about-statistics/presentation/index.php

Presenting Numerical Data

https://www2.le.ac.uk/offices/ld/resources/numerical-data/numerical-data
What is Program Evaluation 38

• Free-to-use Statistical Software

Links to sites with free to use software

https://sites.google.com/site/gsocialchange/statsoft
and
http://statpages.info/javasta2.html
and
http://www.bettycjung.net/Statpgms.htm

Examples:

R https://cran.r-project.org

Epi Info https://www.cdc.gov/epiinfo/index.html

jamovi https://www.jamovi.org

JASP https://jasp-stats.org/

PSPP http://www.gnu.org/software/pspp/

SAS University Edition https://www.sas.com/en_us/software/university-edition.html

What is Program Evaluation 39

Putting it all together

In sum, planing a program evaluation includes answering these three key points:

1. What, exactly, is the question?

2. How will you get the information to answer the question?
3. What will you do with the answers to the question?
What is Program Evaluation 40

Additional Resources

Books/Guides

The Program Manager's Guide to Evaluation, 2010. Office of Planning, Research and Evaluation. Administration for Children and
Families. US Department of Health and Human Services
https://www.acf.hhs.gov/opre/research/project/the-program-managers-guide-to-evaluation

Introduction to program evaluation for public health programs: A self-study guide. CDC. https://www.cdc.gov/eval/guide/index.htm

Making evaluations matter: a practical guide for evaluators

Kusters, C.S.L.; Vugt, S.M. van; Wigboldus, S.A.; Williams, B.; Woodhill, A.J. Wageningen University. 2011
http://library.wur.nl/WebQuery/wurpubs/405451

Community Toolbox, Chapters 36-39, Evaluating Community Programs and Initiatives. http://ctb.ku.edu/en/table-of-contents

What makes a good evaluation. Devina Buchkshee. 6/12/2017. 4th Wheel Social Impact.
http://the4thwheel.com/blog/2017/12/06/what-makes-a-good-evaluation/

Organizations/Sites about improving evaluation

EvalPartners
https://www.evalpartners.org
foster knowledge sharing and networking among M&E practitioners worldwide

Genuine Evaluation.
http://genuineevaluation.com
Patricia J Rogers and E Jane Davidson blog about real, genuine, authentic, practical evaluation

Better Evaluation
http://www.betterevaluation.org
An international collaboration to improve evaluation practice and theory by sharing and generating information about options
(methods or processes) and approaches.
What is Program Evaluation 41

Links to more resources

Betty C. Jung's Evaluation Resources on the Internet

http://www.bettycjung.net/Evaluation.htm

Free Resources on the Web

https://sites.google.com/site/gsocialchange/

Lars Balzer’s Evaluation Portal

http://www.evaluation.lars-balzer.name/links/
What is Program Evaluation 42

Copyright by Gene Shackman, July 2008, February 2010, November 2017, January 2018, February 2018

Some usual legal disclaimers:

This guide can be freely distributed without need for permission, provided it is distributed as is. Distribution for any commercial purpose is strictly
forbidden. This guide cannot be sold under any circumstances. I absolutely do NOT give permission for any of this to be quoted without
attribution on wikipedia.

This guide is only for education purposes. It does not represent any guidelines, recommendations or requirements about how to do program
evaluation. The only purpose is to provide the general public, consumers, students, and evaluators with information about things that may go into
evaluations, so that evaluation may be better understood, and evaluators and clients might work better together to get more out of their evaluation.

In my work on this guide, I do not represent or speak for any organization. I prepared this on my own time, at home, and was not supported by any
organization.

I also benefited greatly from feedback from folks on various email lists.

Materials on web sites listed in this guide do not necessarily reflect my opinions, nor do I assume any responsibility for the content provided at
these web sites. This guide only lists web sites with legal content. Listing a website is not necessarily endorsement of any services or organization.
The sites are only listed because they have some freely available information. I also do not have any financial relationships with any site or
organization listed on this guide.

This guide does not contain or promote; pornography, hatred, racism, propaganda, warez, nudity, hacking activities, violence, degradation, harm
or slander. To the best of my knowledge, I do not list any website that contains or promotes any of that either.