A Basic Guide to Program Evaluation

By Carter McNamara
There are numerous books and other materials that provide in-depth analysis
of evaluations, their designs, methods, combination of methods and
techniques of analysis. But you don't have to be an expert in these topics to
carry out a useful program evaluation. The "20-80" rule applies here20
percent of the effort generates 80 percent of the results. It's better to do a
simple evaluation than to do no evaluation at all. (And if you decide to bring
in an evaluation consultant, make sure you're a smart consumer. Far too
many program evaluations generate information that is either impractical or
irrelevantif the information is understood at all.)
Let's start by examining some common myths about program evaluation:
Myth #1 - "Evaluation is an extraneous activity that generates lots of
boring data with useless conclusions."
That may have been a problem in the past, when program evaluation
methods were chosen largely on the basis of their ability to achieve complete
scientific accuracy, reliability and validity. That approach would often
generate extensive data from which very carefully chosen conclusions would
be drawn. Generalizations and recommendations were avoided. As a result,
evaluation reports tended to reiterate the obvious and left program
administrators disappointed and skeptical about the value of evaluation in
general. More recently (especially as a result of Michael Patton's pioneering
work with utilization-focused evaluation techniques), evaluation has focused
on utility, relevance and practicality at least as much as scientific validity.
Myth #2 - "Evaluation is about proving the success or failure of a
program."
This myth assumes that success is implementing the perfect program and
never having to hear from employees, customers or clients againthe
program will now run itself perfectly. That doesn't happen in real life.
Success involves remaining open to continuing feedback and adjusting the
program accordingly. Evaluation gives you this continuing feedback.

Page 1 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com

Myth #3 - "Evaluation is a unique and complex process that occurs at

a certain time in a certain way, and almost always includes the use
of outside experts."
Many people believe that they must be completely conversant with concepts
such as validity and reliability. They don't. They do have to consider what
information they will require in order to make current decisions about
program issues or needs. And they have to be willing to commit to
understanding what is really going on. People regularly undertake some form
of program evaluationthey just don't do it in a formal fashion, so they don't
get the most out of their efforts. Or they draw conclusions that are
inaccurate.
So just what is program evaluation and what can it legitimately be expected
to accomplish? Program evaluation can help us to:
1. Understand, verify or increase the impact of products or services
on customers or clients. These "outcome" evaluations are increasingly
required by nonprofit funders as a means of verifying that grantees are
indeed helping their constituents. Too often, service providers (for-profit as
well as nonprofit) depend on their own instincts and passions to decide what
their customers or clients really need and whether the products or services
they provide are in fact what is needed. Over time, these organizations find
themselves relying on guesswork to decide what would be a good product or
service, and relying on trial and error to decide how new products or services
should be delivered.
2. Improve delivery mechanisms so that they will be more efficient
and less costly. Over time, product or service delivery can easily turn into a
collection of activities that are less efficient and more costly than they need to
be. Evaluations can identify program strengths and weaknesses to improve
the program.
3. Verify that you're doing what you think you're doing. Typically,
plans about how to deliver services end up changing substantially as those
plans are implemented. Evaluations can verify whether the program is really
running as it was originally planned.
4. Facilitate management's thinking about what a program is all
about. This includes thinking about the program's goals, how to reach its
goals, and how to know whether it has reached its goals.

Page 2 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com

5. Produce data or verify results that can be used for public relations
and promoting services in the community.
6. Produce valid comparisons between programs to decide which
should be retained. This is especially important when the organization has
to cut programs because of budget cuts.
7. Fully examine and describe effective programs for replication
elsewhere.
Usually, management is faced with having to make major decisions due to
decreased funding, ongoing complaints, unmet needs among customers and
clients, or the need to improve service delivery. Your program evaluation
plans depend on what information you must collect in order to make such
decisions. Do you want to know more about what is actually going on in your
programs, whether a particular program is meeting its goals, or about the
impact of a program on customers or clients? You may want your evaluation
to yield one of these results, a combination of these results, or other
information. Ultimately, it's up to you.
The more focused you are about what you want to examine in carrying out
the evaluation, the more efficient you can be, the less time it will take, and
the less it will probably cost you.
There are trade-offs, too, in the breadth and depth of information you get.
The more breadth you want, usually the less depth you get (unless you have
lots of resources at your disposal). On the other hand, if you want to examine
a certain aspect of a program in great detail, you will probably not get as
much information about other aspects of the program.
Consider the following key questions when designing a program evaluation.
1. For what purposes is the evaluation being done? In other words,
what do you want to be able to decide as a result of the evaluation?
2. Who are the audiences for the information from the evaluation?
(e.g., funders, board, management, staff, clients)
3. What do you need in order to make an informed decision and/or
enlighten your intended audiences? (e.g., information to explain the
product or programits inputs, activities and outputs; the customers or
Page 3 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com

clients who use the product or program; strengths and weaknesses of the
product or program; outcomes for customers or clients; how and why the
product or program failed)
4. From what sources should the information be collected? (e.g.,
employees, customers, clients, groups of customers or clients and employees
together, program documentation)
5. How can that information be collected? (e.g., questionnaires,
interviews, examining documentation, observing customers or employees,
conducting focus groups of customers or employees)
6. When is the information needed and by when must it be collected?
7. What resources are available to collect the information?
Goals-Based Evaluation
Often programs are established to meet one or more specific goals. Goalsbased evaluations involve evaluating the extent to which programs are
meeting predetermined goals or objectives. Here are questions to ask yourself
when designing an evaluation to see if you have reached your goals:
1. How were the program goals (and objectives, if applicable)
established? Was the process effective?
2. What is the status of the program's progress toward achieving the
goals?
3. Will the goals be achieved according to the timelines specified in
the program implementation or operations plan? If not, why not?
4. Do personnel have adequate resources (money, equipment,
facilities, training, etc.) to achieve the goals?
5. How should priorities be changed to put more focus on achieving
the goals? (Depending on the context, this might be viewed as a program
management decision, more than an evaluation question.)

Page 4 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com

6. How should timelines be changed? (Be careful about making these

changesknow why efforts are behind schedule before timelines are
changed.)
7. How should goals be changed? (Be careful about making these
changesknow why efforts are not achieving the goals before changing the
goals. And ask whether any goals should be added or removed.)
8. How should goals be established in the future?
Process Evaluation
Process evaluations are geared to fully understanding how a program
workshow it produces the results that it does. These evaluations are useful
when long-standing programs have changed over the years, when employees
or customers report numerous complaints about the program, or when there
appear to be inefficiencies in delivering program services. They are also
useful for accurately portraying to outside parties how a program truly
operates (e.g., for replication elsewhere).
Examples of questions to ask yourself when designing a process evaluation
are:
1. On what basis do employees and/or customers decide that
products or services are needed?
2. What is required of employees in order to deliver the product or
services?
3. How are employees trained to deliver the product or services?
4. How do customers or clients come into the program?
5. What is required of customers or clients?
6. How do employees select which products or services will be
provided to the customer or client?
7. What is the general process that customers or clients go through
with the product or program?
Page 5 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com

8. What do customers or clients consider to be strengths of the

product or program?
9. What do staff members consider to be strengths of the product or
program?
10. What typical complaints are heard from employees and/or
customers?
11. What do employees and/or customers recommend to improve the
product or program?
12. On what basis do employees and/or customers decide that the
product or service is no longer needed?
Outcomes Evaluation
Program evaluation with a focus on outcomes has become increasingly
important for nonprofits, and funders are demanding it more and more. An
outcomes evaluation helps you decide if you are really conducting the right
program activities to bring about the results you believe (or better yet, you've
verified) to be needed by your clients (rather than just engaging in busy
activities which seemed reasonable enough at the time). Outcomes are
benefits to clients from participation in the program. Outcomes are usually
stated in terms of enhanced learning (knowledge, perceptions/attitudes or
skills) or conditions, e.g., increased literacy, self-reliance, etc. Outcomes
should not be confused with program outputs or units of services, e.g., the
number of clients who went through a program.
The United Way of America provides an excellent overview of outcomes
evaluation at its Web site (http://national.unitedway.org/outcomes/). It
includes an introduction to outcomes measurement, a program outcome
model, an explanation of why you should measure outcomes, steps to success
for measuring outcomes, examples of outcomes, and outcome indicators for
various programs. The following discussion summarizes some of that
information.
To perform an outcomes evaluation, you should:

Page 6 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com

1. Identify the major outcomes that you want to examine or verify for
the program under evaluation. You might reflect on your mission (the
overall purpose of your organization) and ask yourself what impacts you will
have on your clients as you work towards your mission. For example, if your
overall mission is to provide shelter and resources to abused women, then ask
yourself what benefits this will have on those women if you effectively
provide them shelter and other services or resources. As a last resort, you
might ask yourself, "What major activities are we doing now?" and then for
each activity ask, "Why are we doing that?" The answer to this "Why?"
question is usually an outcome. This "last resort" approach, though, may just
end up justifying ineffective activities you are doing now, rather than
examining what you should be doing in the first place.
2. Choose the outcomes that you want to examine, prioritize the
outcomes and, if your time and resources are limited, pick the top two or
three most important outcomes to examine for now.
3. For each outcome, specify what observable measures, or
indicators, will suggest that you're achieving that key outcome with
your clients. This is often the most important and enlightening step in
outcomes-based evaluation. However, it is often the most challenging and
confusing step as well, because you're suddenly going from an amorphous
concept (e.g., increased self-reliance) to specific activities (e.g., supporting
clients in getting themselves to and from work). It helps to have a "devil's
advocate" during this phase of identifying indicators, i.e., someone who can
question why you can assume that an outcome was reached because certain
associated indicators were present.
4. Specify a "target" goal of clients, i.e., the number or percentage of
clients with whom you commit to achieving specific outcomes- for example,
"increased self-reliance (an outcome) for 25 percent of adult, African
American women living in Placerville neighborhood of Metropolis as
evidenced by the following measures (indicators)..."
5. Identify what information is needed to show these indicators. For
example, you'll need to know how many clients in the target group went
through the program, how many of them reliably undertook their own
transportation to work, and so on. If your program is new, you may need to
evaluate the process in the program to verify that the program is indeed
carried out according to your original plans.

Page 7 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com

6. Decide how that information can be realistically and efficiently

gathered. Consider program documentation, observation of program
personnel and clients in the program, questionnaires and interviews about
clients' perceived benefits from the program, case studies of program failures
and successes, etc.
7. Analyze and report the findings.

Carter McNamara, MBA, PhD, is co-founder of Authenticity Consulting,

LLC, which specializes in management and organizational development
services for nonprofits. He is also founder and developer of the Free
Management Library, the most extensive collection of online management
resources for nonprofit organizations. This article is excerpted from his Field
Guide to Nonprofit Program Design, Marketing and Evaluation, one of four
volumes that comprise Authenticity Consulting's Nonprofit Capacity Building
Kit. Copyright 2002 Authenticity Consulting, LLC. For more information
about Authenticity Consulting's services and products, or to access the Free
Management Library, visit http://www.authenticityconsulting.com.

Page 8 of 8
Copyright 2002 Authenticity Consulting, LLC.
This article may not be reprinted, reproduced, or retransmitted in whole or in part without
the express written consent of the author.
Reprinted here by permission given to The Grantsmanship Center.
(800) 421-9512
Join Our Mailing List
http://www.tgci.com