Introduction

Program evaluation is a resource-intense activity that can provide program managers with a
great deal of information about how well their programs are operating. Unfortunately it is not
uncommon for an evaluator to discover after an evaluation is underway that the program is not
ready for it. A program may not be fully operational, for example, or may not be able to handle
the data collection requirements of an evaluation. Determining whether a program is ready for
an evaluation prior to beginning it can help ensure that precious evaluation resources are used
at the most appropriate time.

There are two main types of program evaluation—process and outcome. A process evaluation
focuses on program implementation and operation. It identifies the procedures and the
decisions made in developing the program, and it describes how the program operates, the
services it delivers, and the functions it carries out. Outcome evaluation is used to identify the
results of a program’s effort. It seeks to answer the question, “What difference did the program
make?” In order for either of these types of evaluation to be carried out successfully, it is
important that a program be ready for evaluation. One way to determine its readiness is to have
an evaluator conduct an Evaluability Assessment (EA). Developed by Joseph Wholey in 1979, EA
is a tool that can help an evaluator determine whether a program meets the criteria for a
meaningful evaluation to take place.

The purpose of this briefing is to introduce program managers to the concept of Evaluability
Assessment. Even if a program does not undergo a formal EA, the concepts and ideas are
nevertheless important for a program manager to understand and consider prior to having an
evaluation conducted. Program managers should bear in mind how these concepts may affect
the evaluation process and results.

What Is Evaluability Assessment?

Evaluability Assessment (EA) is a systematic process that helps identify whether program
evaluation is justified, feasible, and likely to provide useful information. It not only shows
whether a program can be meaningfully evaluated, but also whether conducting the evaluation
is likely to contribute to improved program performance and management. An evaluator needs
to answer some important questions about a program before a process and/or outcome
evaluation takes place. These questions are the focus of this briefing

EA is a formal process that requires knowledge of and commitment to the program. It can take
several weeks to complete. While EAs can be conducted by a program staff member who is
knowledgeable about evaluation, they are probably most successfully conducted by a
professional evaluator. (The second briefing in this series, Hiring and Working With an Evaluator,
discusses the benefits of internal vs. external evaluation at length.) It is important to keep in
mind that the person performing the EA should have a strong background in evaluation and
juvenile justice. The program’s documents and case files should be ready for review, and staff
should be ready to be interviewed. It is necessary for an evaluator to look at all of these
program pieces to fully understand what a program does on a day-to-day basis.
Although an EA may sound very similar to a process evaluation, there are some important
differences between the two. As stated above, an EA is a systematic process that helps identify
whether program evaluation is justified, feasible, and likely to provide useful information. An EA
determines whether a program is ready for evaluation—either a process or outcome evaluation,
or both. Conducting an EA can tell the evaluator whether the program is able to produce the
information required for a process evaluation, and whether the program meets the other
criteria for beginning an outcome evaluation. Since process evaluations require a great deal of
time and effort, it is important to determine whether the program is ready to undergo such an
evaluation. There are similarities in the nature of the data collected in an EA and a process
evaluation. EA, however, determines whether a program has the basic foundation for an
evaluation to take place (data collection, program model, adequate staffing, etc.), while process
evaluation takes the components that make up the foundation and assesses whether and how
they are utilized

EA is a stepping-stone toward any type of evaluation, whether it is a large process or outcome

evaluation or a smaller, internal assessment of program performance. While the EA is taking
place, the evaluator is also working with program staff, funding agencies, administrators, and
participants to help the program get ready for an evaluation. For example, he/she can help
clarify program goals by making them more realistic and meaningful. This is a major advantage
of having an EA— it will improve a future evaluation by formalizing the agreement between the
evaluator and decisionmakers on what is important in the program, anticipating evaluation
problems, and smoothing the overall process.

Is Your Program Ready For Evaluation?

One possible outcome of an EA is the conclusion that a program is not ready for an evaluation.
Most often the reasons are related to shortcomings in the program’s design or implementation.

Program Design Issues

Two major flaws can make a program unevaluable:

1. No formal program design or model is in place. As discussed in Juvenile Justice Program

Evaluation: An Overview, the first briefing in this series, a program needs to have a design or
model that lays out its goals and objectives, as well as their relationship to program activities.
Without these elements, the program cannot be effectively assessed. A program model must be
in place so that what is actually happening in the program can be compared to what the
program was designed to achieve.

2. The program design or model is unsound. Programs are designed to address specific needs or
to solve specific problems. In order to address these needs or problems, programs must develop
realistic and achievable goals, plausible objectives that can be measured, and activities related
to those objectives. If goals or objectives are unrealistic or unattainable, or activities are
unrelated to objectives, then the program cannot succeed, and the evaluation is a waste of time
and resources. Unattainable goals and objectives might result from the program proposing to
change an entire community’s behavior, for example, while providing direct services to a
relatively few juveniles in the community. Activities that are unrelated to goals and objectives
might result from the program designers misunderstanding the causes of the behaviors they are
targeting for change. In any case, having an unsound program design renders the program
unevaluable in much the same way as having no program design at all.

Program Implementation Issues

Similar to a process evaluation, a major task of an EA is to compare the program design to the
program in operation. If program operation varies greatly from its original design, it will be
impossible for the evaluator to attribute the outcomes of the program to the program itself. If
this is discovered early on in an EA, time and money will be saved. In the EA, the following
questions about program implementation are importan

1. Does the program serve the population for whom it was designed? Even though a program
may be up and running, it may not be serving the type of population it set out to serve, and an
evaluation concerning program success would be misleading. For example, consider a new
restorative justice program for property offenders. One of the program’s objectives is to ensure
that these juveniles provide restitution to the victim. The program has been running for about a
year, and has been serving many youth. However, when looking at the program’s data on the
youths being served, it becomes apparent that the judge has been sending drug offenders to the
program, not property offenders. Even though the program has spent the last year serving
youth, it has not been functioning as intended. This is not necessarily the fault of the program,
but it still would be a waste of time and money to have an evaluator try to conduct a process or
outcome evaluation on the impact of the program for property offenders. The program would
be better off finding out about this major problem through an EA rather than through a more
expensive and timeconsuming evaluation.

2. Does the program have the resources discussed in the program design? In order for a
program to function well, it requires resources, such as well-trained staff, equipment, and space.
If any of the required resources are not present, problems with implementation arise. For
example, if a program has an insufficient number of staff to run the program or if the staff
members do not have the appropriate background qualifications and training, it will be difficult
for the program to achieve its objectives

3. Are the program activities being implemented as designed? Even though a program may be
well designed, the design must be carried out as planned in order for the evaluation to be able
to attribute outcomes to the program itself. Since the evaluation will collect data based on the
actual activities implemented, substantial differences between the stated design and the actual
activities will mean that the evaluation will be assessing the program as imple-

mented, not as planned. If the program activities being implemented are very different from
those planned, then what is being E evaluated is essentially a different program from the one
that was initially proposed. Though the evaluation can assess the activities as implemented, it
cannot assess the program itself, since the logical connection between activities and goals and
objectives has been broken. In this case, the program design should be respecified before the
evaluation takes place.
4. Does the program have the capacity to provide data for an evaluation? Both process and
outcome evaluations require programs to produce a great deal of data. It is crucial that
programs document the activities of their clients and staff members and the services they
provide using forms such as intake assessments, progress reports, and other formal records. In
addition, programs must develop measures to assess their progress in achieving their goals and
objectives, and must systematically measure changes in the juveniles they serve. The program
must have in place, or have the capacity to develop, procedures to generate the data that would
be required for the evaluation. The evaluation cannot commence until these data collection
procedures are in place