PPA 502 – Program

Evaluation
Lecture 8b – Basic Guide to
Program Evaluation
 Source
• Basic Guide to Program Evaluation
– Written by Carter McNamara, MBA, PhD,
Authenticity Consulting, LLC. Copyright 1997-
2006.
– Adapted from the Field Guide to Nonprofit
Program Design, Marketing and Evaluation.
 A Brief Introduction ...
• Note that the concept of program evaluation can
include a wide variety of methods to evaluate
many aspects of programs in nonprofit or for-
profit organizations.
• However, personnel do not have to be experts in
these topics to carry out a useful program
evaluation. The "20-80" rule applies here, that
20% of effort generates 80% of the needed
results. It's better to do what might turn out to be
an average effort at evaluation than to do no
evaluation at all.
 Some Myths about Program
Evaluation
• Many people believe evaluation is a
useless activity that generates lots of
boring data with useless conclusions.
• More recently (especially as a result of
Michael Patton's development of
utilization-focused evaluation), evaluation
has focused on utility, relevance and
practicality at least as much as scientific
validity.
 Some Myths about Program
Evaluation
• Many people believe that evaluation is about
proving the success or failure of a program.
• This myth assumes that success is
implementing the perfect program and never
having to hear from employees, customers or
clients again -- the program will now run itself
perfectly.
• This doesn't happen in real life.
• Success is remaining open to continuing
feedback and adjusting the program accordingly.
• Evaluation gives you this continuing feedback.
 Some Myths about Program
Evaluation
• Many believe that evaluation is a highly unique and
complex process that occurs at a certain time in a certain
way, and almost always includes the use of outside
experts.
• Many people believe they must completely understand
terms such as validity and reliability.
• They don't have to.
• They do have to consider what information they need to
make current decisions about program issues or needs.
• And they have to be willing to commit to understanding
what is really going on.
So What is Program Evaluation?
• First, we'll consider "what is a program?"
• Typically, organizations work from their mission to
identify several overall goals which must be reached to
accomplish their mission.
• In public agencies and nonprofits, each of these goals
often becomes a program.
• Public and nonprofit programs are organized methods to
provide certain related services to constituents, e.g.,
clients, customers, patients, etc.
• Programs must be evaluated to decide if the programs
are indeed useful to constituents.
So What is Program Evaluation?
• Program evaluation is carefully collecting
information about a program or some aspect of
a program in order to make necessary decisions
about the program.
• Don't worry about what type of evaluation you
need or are doing -- worry about what you need
to know to make the program decisions you
need to make, and worry about how you can
accurately collect and understand that
information.
 Where Program Evaluation is
Helpful
• Understand, verify or increase the impact
of products or services on customers or
clients.
• Improve delivery mechanisms to be more
efficient and less costly.
• Verify that you're doing what you think
you're doing.
 Other Reasons to Conduct
Evaluations
• Facilitate management's really thinking
about what their program is all about,
including its goals, how it meets it goals
and how it will know if it has met its goals
or not.
• Produce data or verify results that can be
used for public relations and promoting
services in the community.
 Other Reasons to Conduct
Evaluations
• Produce valid comparisons between
programs to decide which should be
retained, e.g., in the face of pending
budget cuts.
• Fully examine and describe effective
programs for duplication elsewhere.
 Basic Ingredients: Organization
and Program(s)
• You Need An Organization:
– This may seem too obvious to discuss, but before an
organization embarks on evaluating a program, it
should have well established means to conduct itself
as an organization.
• You Need Program(s):
– To effectively conduct program evaluation, you should
first have programs. That is, you need a strong
impression of what your customers or clients actually
need.
– Next, you need some effective methods to meet each
of those goals. These methods are usually in the form
of programs.
 Programs
• Inputs are the various resources needed
to run the program, e.g., money, facilities,
customers, clients, program staff, etc.
• The process is how the program is carried
out.
 Programs
• The outputs are the units of service, e.g.,
number of customers serviced, number of clients
counseled, children cared for, artistic pieces
produced, or members in the association.
• Outcomes are the impacts on the customers or
on clients receiving services, e.g., increased
mental health, safe and secure development,
richer artistic appreciation and perspectives in
life, increased effectiveness among members,
etc.
 Planning Your Program
Evaluation
• Depends on what information you need to
make your decisions and on your resources.
– Your program evaluation plans depend on what
information you need to collect in order to make
major decisions.
– But the more focused you are about what you want to
examine by the evaluation, the more efficient you can
be in your evaluation, the shorter the time it will take
you and ultimately the less it will cost you.
 Key Considerations
• For what purposes is the evaluation being
done, i.e., what do you want to be able to
decide as a result of the evaluation?
• Who are the audiences for the information
from the evaluation, e.g., customers,
bankers, funders, board, management,
staff, customers, clients, etc.
 Key Considerations
• What kinds of information are needed to make
the decision you need to make and/or enlighten
your intended audiences, e.g., information to
really understand the process of the product or
program (its inputs, activities and outputs), the
customers or clients who experience the product
or program, strengths and weaknesses of the
product or program, benefits to customers or
clients (outcomes), how the product or program
failed and why, etc.
 Key Considerations
• From what sources should the information be
collected, e.g., employees, customers, clients,
groups of customers or clients and employees
together, program documentation, etc.
• How can that information be collected in a
reasonable fashion, e.g., questionnaires,
interviews, examining documentation, observing
customers or employees, conducting focus
groups among customers or employees, etc.
 Key Considerations
• When is the information needed (so, by
when must it be collected)?
• What resources are available to collect the
information?
 Goals-Based Evaluation
1. How were the program goals (and objectives, is
applicable) established? Was the process
effective?
2. What is the status of the program's progress
toward achieving the goals?
3. Will the goals be achieved according to the
timelines specified in the program
implementation or operations plan? If not, then
why?
4. Do personnel have adequate resources
(money, equipment, facilities, training, etc.) to
achieve the goals?
 Goals-Based Evaluation
5. How should priorities be changed to put more focus on
achieving the goals? (Depending on the context, this
question might be viewed as a program management
decision, more than an evaluation question.)
6. How should timelines be changed (be careful about
making these changes - know why efforts are behind
schedule before timelines are changed)?
7. How should goals be changed (be careful about making
these changes - know why efforts are not achieving the
goals before changing the goals)? Should any goals be
added or removed? Why?
8. How should goals be established in the future?
 Process-Based Evaluations
• 1. On what basis do employees and/or the
customers decide that products or
services are needed?
• 2. What is required of employees in order
to deliver the product or services?
 Process-Based Evaluations
• 3. How are employees trained about how
to deliver the product or services?
• 4. How do customers or clients come into
the program?
• 5. What is required of customers or client?
 Process-Based Evaluations
• 6. How do employees select which
products or services will be provided to the
customer or client?
• 7. What is the general process that
customers or clients go through with the
product or program?
• 8. What do customers or clients consider
to be strengths of the program?
 Process-Based Evaluations
• 9. What do staff consider to be strengths
of the product or program?
• 10. What typical complaints are heard
from employees and/or customers?
 Process-Based Evaluations
• 11. What do employees and/or customers
recommend to improve the product or
program?
• 12. On what basis do employees and/or
the customer decide that the product or
services are no longer needed?
 Outcome-Based Evaluations
• Identify the major outcomes that you want to
examine or verify for the program under
evaluation.
• Choose the outcomes that you want to examine,
prioritize the outcomes and, if your time and
resources are limited, pick the top two to four
most important outcomes to examine for now.
• For each outcome, specify what observable
measures, or indicators, will suggest that you're
achieving that key outcome with your clients.
 Outcome-Based Evaluations
• Specify a "target" goal of clients, i.e., with
what number or percent of clients you
commit to achieving specific outcomes?
• Identify what information is needed to
show these indicators.
• Decide how can that information be
efficiently and realistically gathered.
• Analyze and report the findings.
 Methods to Collection Information

• Questionnaires, surveys, checklists.

• Interviews.
• Documentation review.
• Observation.
• Focus groups.
• Case studies.
Methods to Collection Information
Methods to Collection Information
Methods to Collection Information
 Ethics: Informed Consent from
Program Participants
• Note that if you plan to include in your
evaluation, the focus and reporting on personal
information about customers or clients
participating in the evaluation, then you should
first gain their consent to do so.
• They should understand what you're doing with
them in the evaluation and how any information
associated with them will be reported.
 Ethics: Informed Consent from
Program Participants
• You should clearly convey terms of
confidentiality regarding access to
evaluation results.
• They should have the right to participate or
not.
• Have participants review and sign an
informed consent form.
Selecting Which Methods to Use
• The overall goal in selecting evaluation method(s) is to
get the most useful information to key decision makers in
the most cost-effective and realistic fashion.
• Note that, ideally, the evaluator uses a combination of
methods, for example, a questionnaire to quickly collect
a great deal of information from a lot of people, and then
interviews to get more in-depth information from certain
respondents to the questionnaires. Perhaps case studies
could then be used for more in-depth analysis of unique
and notable cases, e.g., those who benefited or not from
the program, those who quit the program, etc.
Selecting Which Methods to Use
1. What information is needed to make current
decisions about a product or program?
2. Of this information, how much can be collected
and analyzed in a low-cost and practical
manner, e.g., using questionnaires, surveys and
checklists?
3. How accurate will the information be (reference
the above table for disadvantages of methods)?
4. Will the methods get all of the needed
information?
Selecting Which Methods to Use
5. What additional methods should and could be used if
additional information is needed?
6. Will the information appear as credible to decision
makers, e.g., to funders or top management?
7. Will the nature of the audience conform to the methods,
e.g., will they fill out questionnaires carefully, engage in
interviews or focus groups, let you examine their
documentations, etc.?
8. Who can administer the methods now or is training
required?
9. How can the information be analyzed?
 Four Levels of Evaluation:
• There are four levels of evaluation information
that can be gathered from clients, including
getting their:
1. reactions and feelings (feelings are often poor
indicators that your service made lasting impact)
2. learning (enhanced attitudes, perceptions or
knowledge)
3. changes in skills (applied the learning to enhance
behaviors)
4. effectiveness (improved performance because of
enhanced behaviors)
 Four Levels of Evaluation:
• Usually, the farther your evaluation
information gets down the list, the more
useful is your evaluation. Unfortunately, it
is quite difficult to reliably get information
about effectiveness. Still, information
about learning and skills is quite useful.
 Analyzing and Interpreting
Information
• Always start with your evaluation
goals:
– When analyzing data (whether from
questionnaires, interviews, focus groups, or
whatever), always start from review of your
evaluation goals, i.e., the reason you
undertook the evaluation in the first place.
This will help you organize your data and
focus your analysis.
 Analyzing and Interpreting
Information
• Basic analysis of "quantitative" information (for
information other than commentary, e.g.,
ratings, rankings, yes's, no's, etc.):
1. Make copies of your data and store the master copy
away. Use the copy for making edits, cutting and
pasting, etc.
2. Tabulate the information, i.e., add up the number of
ratings, rankings, yes's, no's for each question.
3. For ratings and rankings, consider computing a
mean, or average, for each question.
4. Consider conveying the range of answers, e.g., 20
people ranked "1", 30 ranked "2", and 20 people
ranked "3".
 Analyzing and Interpreting
Information
• Basic analysis of "qualitative" information
(respondents' verbal answers in interviews,
focus groups, or written commentary on
questionnaires):
1. Read through all the data.
2. Organize comments into similar categories.
3. Label the categories or themes, e.g., concerns,
suggestions, etc.
4. Attempt to identify patterns, or associations and
causal relationships in the themes.
5. Keep all commentary for several years after
completion in case needed for future reference.
 Interpreting Information:
1. Attempt to put the information in perspective.
a) Compare results to what you expected;
b) Management or program staff
c) Common standards for your services
d) Original program goals (program evaluation)
e) Indications of accomplishing outcomes (outcomes evaluation)
f) Description of the program's experiences, strengths,
weaknesses, etc. (process evaluation).
2. Consider recommendations to help program staff
improve the program, conclusions about program
operations or meeting goals, etc.
3. Record conclusions and recommendations in a report
document, and associate interpretations to justify your
conclusions or recommendations.
Reporting Evaluation Results
1.The level and scope of content depends
on to whom the report is intended,
2.Be sure employees have a chance to
carefully review and discuss the report.
Translate recommendations to action
plans, including who is going to do what
about the program and by when.
Reporting Evaluation Results
3. Bankers or funders will likely require a report that
includes an executive summary (this is a summary of
conclusions and recommendations, not a listing of what
sections of information are in the report -- that's a table
of contents); description of the organization and the
program under evaluation; explanation of the evaluation
goals, methods, and analysis procedures; listing of
conclusions and recommendations; and any relevant
attachments,
4. Be sure to record the evaluation plans and activities in
an evaluation plan which can be referenced when a
similar program evaluation is needed in the future.
 Pitfalls to Avoid
1. Don't balk at evaluation because it seems far
too "scientific." It's not. Usually the first 20% of
effort will generate the first 80% of the plan, and
this is far better than nothing.
2. There is no "perfect" evaluation design. Don't
worry about the plan being perfect. It's far more
important to do something, than to wait until
every last detail has been tested.
 Pitfalls to Avoid
3. Work hard to include some interviews in your evaluation
methods. Questionnaires don't capture "the story," and
the story is usually the most powerful depiction of the
benefits of your services.
4. Don't interview just the successes. You'll learn a great
deal about the program by understanding its failures,
dropouts, etc.
5. Don't throw away evaluation results once a report has
been generated. Results don't take up much room, and
they can provide precious information later when trying
to understand changes in the program.