Basic Principles of Program Evaluation

Corinne Datchi-Phillips, Ph.D. CEBP Learning Institute May 26th , 2010

Breakout Session Objectives

1. Participants will increase their understanding of the purpose and goals of program evaluation. 2. Participants will be able to identify what questions they may ask to improve their correctional programming. 3. Participants will become knowledgeable about the goals of different types of program evaluation. 4. Participants will learn about existing resources they may use to guide the evaluation of their services. 5. Participants will be introduced to the recommendations of the CEBP regarding data collection in program evaluation.

6. 7.

Basic Principles of Program Evaluation

The goal and purpose of program evaluation

Program Evaluation
Purpose
To provide administrators with information about program activities and outcomes To contribute to the delivery of quality service to people in need

Program Evaluation
Goals:
Improve correctional treatment and supervision programs (enhance effectiveness and quality of service); Inform program-level decisions; Help public officials meet the obligation of using resources effectively by identifying and providing cost-effective programs.

Program Evaluation
Goals
To inform decisions by identifying
The need for specific services The likelihood that these services will be utilized What amount of service is necessary to meet the needs that have been identified

Program Evaluation
Goals
To inform decisions about the selection and implementation of correctional treatment and supervision programs
Whether to continue offering a program if it is successful Whether to discontinue a program because it fails to meet a need in the community How to alter a program to make it more useful to participants (Continuous quality improvement)
What are the limitations of the program? How can these limitations be addressed?

Program Evaluation
Important questions about program implementation:
Is a program implemented as planned?
Deviations from the original plan of a program may have harmful effects or may result in no or minimal benefit.

What are the outcomes of a program?

Reduction of criminogenic needs Increased public safety (reduction of recidivism rates)

Program Evaluation
Important question about resource utilization:
What programs produce the most favorable outcomes?
When programs are equally effective with this type of client with this type of problem in this type of setting, the least expensive program may be selected.

Basic Principles of Program Evaluation

Common types of program evaluation

Program Evaluation
Different types
1.Need Evaluation 2.Implementation Evaluation 3.Outcome Evaluation 4.Cost-effectiveness Analysis

Need Evaluation
Basic first step that precedes program planning and implementation Goal
To identify and measure critical unmet needs within a community
Nature and magnitude of social, psychological, and behavioral problems Community-based resources and services already available to address the problems

Implementation Evaluation
After a program has been developed and begun
What parts of the program have been implemented or not What are barriers to implementation Who is being served by the program Match between program implementation and program plan

Outcome Evaluation
After satisfactory assessment of program implementation
Different levels of complexity:
Are program participants performing or functioning better ? Are program participants performing or functioning better than those not receiving the services? Does the program cause a positive change in the performance or functioning of the participants? (Causal relationships are more difficult
to establish)

Cost-effectiveness Analysis
After it has been established that a program has been implemented effectively with good outcomes
Cost-effectiveness analysis answers the question of efficiency.
A successful program that requires a large amount of resources may not be a viable option. Among the programs that have a similar positive impact on a specific social and behavioral problem, which ones cost less?
Only compare programs that are designed to achieve the same outcomes with the same group of clients

Basic Principles of Program Evaluation

Resources

Resources
Robert Wood Johnson Foundations Guide to Evaluation Primers
http://www.rwjf.org/pr/product.jsp?id=18657

W.K. Kellogg Foundations Evaluation Handbook

http://www.wkkf.org/knowledge-center/resourc

Resources
The Evaluation Exchange periodical, Harvard University Center for Disease Controls Framework for Program Evaluation in Public Health
http://www.cdc.gov/eval/framework.htm

http://www.hfrp.org/evaluation/the-evaluation-e

Basic Principles of Program Evaluation

Integrating program evaluation with routine practices: The importance of ongoing data collection

Integrating evaluation into routine practices

Program evaluation as a part of program management
Requires the consistent, ongoing collection and analysis of information that will inform program decision making
Detailed description of the program (target population; problem to be addressed; goals and objectives; program activities; mechanisms of change) After determining which questions you would like to answer about a specified program, identify what information you will need to answer these questions Develop a plan for collecting the information
Collect the information from multiple sources and perspectives (clients, staff, documents, observations)

Data Gathering
CEBP recommendations (2009)
Basic data elements necessary to perform program outcome evaluations

A survey of current community corrections practices in the IDOC, recommendations p. 57, Appendix B p. 71 http://education.indiana.edu/Default.aspx?alia

Data Gathering
CEBP recommendations (2009)
Basic data elements that will help answer the question what works? (effects of community corrections programs)
With whom?
Type and severity of problems Human differences (age, sex, etc.) Cultural differences

Under what conditions?

Treatment dosage Combination of services/programs

Data Gathering
CEBP recommendations (2009)
Specification of community corrections programming
Target population; problem to be addressed; goals and objectives; program activities; mechanisms of change

Effectiveness of community corrections programming

Program completion rates (criteria) Measures of behavioral change (improvement in performance and functioning) Recidivism rates

Basic Principles of Program Evaluation

Conclusions & Questions

Program Evaluation
Essential to evaluate correctional practices
To identify practices that are useful, effective, and feasible under particular conditions To improve programming that address the needs of community corrections clients and communities To support accountability in community corrections
Using scientific methods and knowledge to inform decisions and actions

Program Evaluation
Common concerns about program evaluation
Belief that the evaluation will limit providers capacity to be flexible in their implementation of programs
Flexibility within broad structure of the program

Belief that negative evaluation will result in termination of a program

Will more likely inform refinement of the program

Belief that information will be misused (e.g., threat to promotion of staff)

Program evaluation is distinct from individual staff assessment

Program Evaluation
Common concerns about program evaluation
Belief that staffs experiential knowledge will not be taken into consideration
Staffs observations constitute data that can inform questions about program implementation.

Belief that staff and administrators will no longer be able to make decisions about the way the program is offered
Program evaluation provides information that staff and administrators can use to make changes in the way the program is offered to improve delivery.

Program Evaluation
Questions & Comments