Identifying and Selecting

Evidence-Based Interventions
Guidance Document for the Strategic Prevention Framework
State Incentive Grant Program

January 2007
Identifying and Selecting
Evidence-Based Interventions
Guidance Document for the Strategic Prevention Framework
State Incentive Grant Program
Contents
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
A. Background and Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
B. Purpose of the Guidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

II. SPF Implications for Community Planning to Identify

and Select Evidence-Based Interventions. . . . . . . . . . . . . . . . . . . 5
A. Local Needs and Resource Assessment: Key Data Tool to
Guide Community Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
B. The Community Logic Model: Key Conceptual Tool
for Community Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Examples of Community Logic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

III. Using the Community Logic Model and Assessment

Information to Identify Best Fit Interventions. . . . . . . . . . . . . . . 9
A. Establishing Conceptual Fit: Is It Relevant? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
B. Establishing Practical Fit: Is It Appropriate? . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Utility and Feasibility Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

IV. Using Public Resources/Review Processes to Identify

Evidence-Based Interventions and Determine Their
Evidence Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Evidence-Based Interventions and Evidence Status . . . . . . . . . . . . . . . . . . . . 12
SPF Definitions of Evidence-Based Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
A. Using Federal Lists or Registries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
B. Using Peer-Reviewed Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Assessing Elements of Evidence Reported in
Peer-Reviewed Journals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Identifying and Selecting Evidence-Based Interventions iii

 C. Using Guidelines for Documented Evidence of Effectiveness . . . . . . . . . . . 17
SPF SIG Program Guidelines for Documented Effectiveness . . . . . . . . . . . . 18
Examples of Evidence to Support Documented Effectiveness. . . . . . . . . . . 19

V. Summary Process Description:

Selecting Best Fit Prevention Interventions. . . . . . . . . . . . . . . . 21
VI. SPF SIG Program Guidance: Roles and Expectations . . . . . 22
A. Federal Role. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
B. State/Jurisdiction Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
C. Community Role. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Concluding Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

GLOSSARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Table of Figures
Figure 1. Community Logic Model, Outcomes-Based Prevention . . . . . . . . . 6
Figure 1A. Community Logic Model for Preventing Alcohol-
Involved Traffic Crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Figure 1B. Community Logic Model for Preventing Illicit Drug Use . . . . . . . 8
Figure 2. Human Environmental Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Figure 3. Process Description: Selecting Best Fit
Prevention Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

iv Identifying and Selecting Evidence-Based Interventions

Executive Summary
The purpose of this guidance is to assist State and community planners in applying the
Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Strategic Prevention
Framework (SPF) to identify and select evidence-based interventions that address local needs
and reduce substance abuse problems.

Section I. Summarizes the five steps of SAMHSA’s SPF and sets the stage for selecting evidence-
based interventions to include in a comprehensive strategic plan.

Section II. Focuses on two analytic tasks included under the SPF: assessing local needs, resources,
and readiness to act; and developing a community logic model. Explains the importance of these
tasks in community planning to identify the best evidence-based interventions for specific local
needs.

Section III. Details how prevention planners can apply the community logic model to determine
the conceptual fit or relevance of prevention strategies that hold the greatest potential for affect-
ing a substance abuse problem. Also discusses how to examine candidate interventions from the
perspective of practical fit or appropriateness for local circumstances, contexts, and populations.

Section IV. Discusses the importance of strength of evidence in determining whether specific
interventions work. Presents the three definitions of “evidence-based” status provided under the
SPF SIG Program and the challenges of using each one to select prevention interventions. The
three definitions of “evidence-based” status are as follows:

• Inclusion in a Federal List or Registry of evidence-based interventions;

• Being reported (with positive effects) in a peer-reviewed journal; or
• Documentation of effectiveness based on the guidelines listed below.
During 2005, SAMHSA/Center for Substance Abuse Prevention (CSAP) convened an Expert
Workgroup to develop recommendations for evidence-based programming and guidelines to
define documented effectiveness under the SPF SIG Program. Based on the recommendations of
the Expert Workgroup, SAMHSA/CSAP recommends three guidelines for evidence—all of which
need to be demonstrated—to document the effectiveness of complex or innovative interventions
developed locally for a specific population and context. Taken together, the evidence guidelines
for documented effectiveness are the following:

Guideline 1: The intervention is based on a solid theory or theoretical perspective that has
been validated by research;

Guideline 2: The intervention is supported by a documented body of knowledge—a con-

verging of empirical evidence of effectiveness—generated from similar or related interven-
tions that indicate effectiveness; and

Identifying and Selecting Evidence-Based Interventions 1

 Guideline 3: The intervention is judged by a consensus among informed experts to be
effective based on a combination of theory, research and practice experience.“ Informed
experts” may include key community prevention leaders, and elders or other respected
leaders within indigenous cultures.

Section V. Summarizes the process of working through three considerations that determine the
best fit of interventions to include in comprehensive prevention plans:

• Conceptual fit to the logic model: Is it relevant?

• Practical fit to the community’s needs and resources: Is it appropriate?
• Strength of evidence: Is it effective?
Section VI. Discusses the respective roles and expectations for SAMHSA/CSAP and SPF SIG States
and their subrecipient communities, jurisdictions, and federally recognized tribes and tribal
organizations to ensure the identification and selection of best fit evidence-based prevention
interventions for each community.

2 Identifying and Selecting Evidence-Based Interventions

I. Introduction
A. Background and Context
The Substance Abuse and Mental Health Services Administration (SAMHSA) envisions “a life in the
community for everyone” and has as its mission “building resilience and facilitating recovery.”
SAMHSA strives to achieve its mission through programs supported by three goals: accountabili-
ty, capacity, and effectiveness. The Center for Substance Abuse Prevention (CSAP) helps to create
healthy communities. SAMHSA/CSAP helps States to provide resources and assistance to commu-
nities so that communities, in turn, can prevent and reduce substance abuse and related prob-
lems. SAMHSA/CSAP also provides training, technical assistance, and funds to strengthen the
State prevention systems that serve local communities. SAMHSA/CSAP works with States to iden-
tify programs, policies, and practices that are known to be effective in preventing and reducing
substance abuse and related problems.

All of SAMHSA’s mission and goals are driven by strategic planning to align, manage, and account
for priority programs and issues across the three Centers. Chief among SAMHSA’s priorities is the
Strategic Prevention Framework (SPF)—a five-step planning process to guide the work of States
and communities in their prevention activities.

Step 1. Assess population needs (nature of the substance abuse problem, where it occurs,
whom it affects, how it is manifested), the resources required to address the
problem, and the readiness to act;

Step 2. Build capacity at State and community levels to address needs and problems
identified in Step 1;

Step 3. Develop a comprehensive strategic plan. At the community level, the comprehensive
plan articulates a vision for organizing specific prevention programs, policies, and
practices to address substance abuse problems locally;

Step 4. Implement the evidence-based programs, practices, and policies identified in

Step 3; and

Step 5. Monitor implementation, evaluate effectiveness, sustain effective activities, and

improve or replace those that fail.

Throughout all five steps, implementers of the SPF must address issues of cultural competence
and sustainability. Cultural competence is important for eliminating disparities in services and
programs offered to people of diverse racial, ethnic, and linguistic backgrounds, gender and sexu-
al orientations, and those with disabilities. Cultural competence will improve the effectiveness of
programs, policies, and practices selected for targeted populations.

Identifying and Selecting Evidence-Based Interventions 3

 Sustainability of outcomes is a goal established at the outset and addressed throughout each
step of the SPF. Prevention planners at both State and local levels need to build systems and insti-
tutionalize the practices that will sustain prevention outcomes over time, beyond the life of any
specific program.

Under the SPF State Incentive Grant (SIG) Program, prevention planners are specifically required
to select and implement evidence-based interventions. SAMHSA/CSAP recognized that this
requirement necessitates the availability of a broad array of evidence-based interventions and,
further, must allow prevention planners the flexibility to decide which options best fit their local
circumstances. To assist the field in meeting this requirement, SAMHSA/CSAP convened an Expert
Workgroup during 2005 to develop recommendations and guidelines for selecting evidence-
based interventions under the SPF SIG Program.

The Expert Workgroup was composed of nationally-recognized substance abuse prevention

experts from a wide spectrum of academic backgrounds and theoretical research perspectives.
The guidance presented in this document is grounded in the thinking and recommendations of
the SAMHSA/CSAP Expert Workgroup.

B. Purpose of the Guidance

This guidance is directed to prevention planners working through SPF Steps 3 and 4 and to help
them successfully select and implement evidence-based interventions. The guidance lays out an
analytic process with a few key concepts to apply in selecting interventions that are conceptually
and practically fitting and effective.

4 Identifying and Selecting Evidence-Based Interventions

II. SPF Implications for Community Planning to
Identify and Select Evidence-Based Interventions
A. Local Needs and Resource Assessment: Key Data Tool to
Guide Community Planning
Prevention experts agree that substance abuse problems are usually best addressed locally—at
the community level—because they are manifested locally. Yet some prevention approaches may
be most effective when implemented on a larger scale, perhaps through a statewide change in
laws (e.g., change in the alcohol index for driving under the influence). Experts also agree that
substance abuse problems are among the most difficult social problems to prevent or reduce.
Substance abuse problems require comprehensive solutions—a variety of intervention approach-
es directed to multiple opportunities.

The challenge of selecting the optimal mix of strategies is complicated by the limited availability
of public resources on evidence-based interventions. In practice, practitioners seeking to reduce
substance abuse problems will need to put together their own mix of interventions. The mix of
interventions will need to fit the capacity, resources, and readiness of the community and its par-
ticipating organizations. Some interventions in the comprehensive plan will demonstrate evi-
dence of effectiveness using scientific standards and research methodologies, while others will
demonstrate effectiveness based on less standardized or customized assessment. An optimal mix
of strategies will combine complementary and synergistic interventions drawn from different
resources and based on different types of evidence.

The needs and resource assessments in Step 1 will guide development of the comprehensive
plan, from profiling the problem/population and the underlying factors/conditions that con-
tribute to the problem, to checking the appropriateness of prevention strategies to include in the
plan. It is crucial to use local data and information to identify effective strategies that fit local capacity,
resources, and readiness. However, finding local data is often difficult. Creative approaches to data
sources, including the use of proxy measures and information gleaned through focus groups, may be
necessary.

B. The Community Logic Model: Key Conceptual Tool for Community Planning
The community logic model reflects the planning that needs to take place to generate communi-
ty level change. Building the logic model begins with careful identification or mapping of the
local substance abuse problem (and associated patterns of substance use and consequences) to
the factors that contribute to them. Developing the logic model starts with defining the substance
abuse problem, not choosing the solutions, that is, the programs, practices, or policies already decided
upon by States or communities.

Identifying and Selecting Evidence-Based Interventions 5

 Since comprehensive plans combine a variety of strategies, it is important to understand the rela-
tionships between these problems and the factors or conditions that contribute to them. Few
substance abuse problems are amenable to change through direct influence or attack. Rather,
they are influenced indirectly through underlying factors that contribute to the problem and its
initiation, escalation, and adverse consequences.

These factors include the following:

• Risk and protective factors that present themselves across the course of human develop-
ment and make individuals and groups either more or less prone to substance abuse in
certain social contexts.

• Contributing conditions implicated in the development of the problems and conse-

quences associated with substance abuse. Examples may include specific local policies
and practices, community realities, or population shifts.

Identifying risk and protective factors is central to determining the most promising strategies—
programs, practices and policies—for addressing a substance abuse problem and its initiation,
progression, frequency/quantity of use, and consequences of use.

Linking the substance abuse problem to the underlying factors, and ultimately to potentially effective
prevention strategies, requires analysis and a conceptual tool.The logic model in Figure 1 serves as
the conceptual tool to map the substance abuse phenomenon and the factors that drive it.

Figure 1. Community Logic Model, Outcomes-Based Prevention

Substance abuse Risk and Programs,

& related protective policies &
problems factors/conditions practices

Logic models lay out the community substance abuse problem and the key markers leading to
that problem. They represent systematic plans for attacking local problems within a specific con-
text. The community logic model makes explicit the rationale for selecting programs, policies, and
practices to address the community’s substance abuse problem. Used in this way, the logic model
becomes an important conceptual tool for planning a comprehensive and potentially effective preven-
tion effort.

6 Identifying and Selecting Evidence-Based Interventions

Examples of Community Logic Models
The sample community-level logic models in Figures 1A and 1B illustrate the relationships
between an identified substance abuse problem or consequence and the salient risk and protec-
tive factors/conditions that contribute to the problem. Each risk and protective factor/condition,
in turn, highlights an opportunity—or potential point of entry—for interventions that can lead to
positive outcomes in the targeted problem.

While different communities may show similar substance abuse problems, the underlying factors that
contribute most to them will likely vary from community to community. Communities will tailor the
logic model to fit their particular needs, capacities, and readiness to act.

Figure 1A. Community Logic Model for Preventing

Alcohol-Involved Traffic Crashes (15- to 24-year-olds)

Substance Risk and protective Strategies

abuse-related factors/conditions (Examples)
consequences (Examples)
(Example) Retailer education
Easy retail access to
alcohol for youth
Alcohol-involved Compliance checks/sobriety
traffic crashes Low enforcement of checkpoints
alcohol laws
(15- to 24-year-olds) Parent education/parental
Easy social access to alcohol monitoring
Low perceived risk of Youth education programs
alcohol use
Community education
Social norms accepting and/or
encouraging youth drinking Restrictions on advertising
Promotion of alcohol use to youth
(advertising, movies, music, etc.)
Restrictions on
Low or discount pricing “happy hours,” etc.
on alcohol
Other evidence-based
Other factors from the interventions
research literature

Identifying and Selecting Evidence-Based Interventions 7

 Figure 1B. Community Logic Model for Preventing Illicit Drug Use

Substance abuse Risk and protective Strategies

problem factors/conditions (Examples)
(Example) (Examples)
Family/Parenting
Disrupted parent/
Illicit drug use skills training
child relations
Social skills
Alienation from training
pro-social peers
Tutoring
Academic failure
Changing school
Positive school environment climate

Social competence Communication,

decision-making
Other factors from the and problem solving
research literature skills training
Other evidence-
based interventions

8 Identifying and Selecting Evidence-Based Interventions

III. Using the Community Logic Model and
Assessment Information to Identify Best Fit
Interventions
A. Establishing Conceptual Fit: Is It Relevant?

Relevance: If the prevention program, policy, or practice doesn’t address the underlying
risk and protective factors/conditions that contribute to the problem, then the interven-
tion is unlikely to be effective in changing the substance abuse problem or behavior.
The community logic model can be used to guide the identification and selection of types of pro-
grams, practices, and policies for substance abuse prevention that are relevant for a particular
community. Community logic models are tailored to reflect and meet the unique circumstances
of a particular community. SAMHSA/CSAP expects SPF SIG States to develop an epidemiological
profile and create an initial generic logic model. In turn, each community participating in the pro-
gram will tailor the generic logic model to its needs.

Because substance abuse problems are complex, multiple factors and conditions will be implicat-
ed, some more strongly than others. Communities are encouraged to identify a comprehensive
set of interventions directed to their most significant risk and protective factors/conditions and
targeted to multiple points of entry. Figure 2 illustrates the Human Environmental Framework,
one tool available to guide thinking about multiple points of entry for interventions directed to
risk and protective factors across the life span and across social environments, and defining
points of entry for interventions in different life sectors.

The community logic model can be used to check the conceptual fit of interventions to include in the
comprehensive community plan. The logic model screens for the most appropriate types of inter-
ventions for a particular community.

Identifying and Selecting Evidence-Based Interventions 9

 Figure 2. Human Environmental Framework

Environm
Time
ro ader ent
B
nvironmen
ear E ts
N Environm
a r y
en
rim School t

Eco
P

s
tical Influences

Laws

nom
Social Services

Family Peers

and Policies

ic Influence
Person
Religious
Work
Poli

Groups

s
m
Neighborhood

ste
or
W

k Sy
i cial
Media Jud
The Ecology of Human Development:
Cult s Experiments by Nature and Design,
ura l Influence Bronfenbrenner, Urie, 1979, Harvard
University Press, Cambridge, Mass.

This figure depicts social environments or spheres of influence in concentric circles that flare out-
ward, moving progressively away from direct influence on the individual toward increasingly indi-
rect influence, and advancing over time. A comprehensive intervention plan should identify a mix
or layering of interventions that target salient risk and protective factors in multiple contexts
across the life span.

B. Establishing Practical Fit: Is It Appropriate?

Appropriateness: If the prevention program, policy, or practice doesn’t fit the communi-
ty’s capacity, resources, or readiness to act, then the community is unlikely to imple-
ment the intervention effectively.
A second important concept in selecting prevention interventions is practical fit with the capaci-
ty, resources, and readiness of the community itself and the organizations responsible for imple-
menting interventions. Practical fit is assessed through a series of utility and feasibility checks that
grow out of the needs/resource assessment and capacity-building activities conducted in SPF
Steps 1 and 2.

SAMHSA/CSAP encourages practitioners to use their community assessment findings to judge

the appropriateness of specific programs, policies, and practices deemed relevant to the factors

10 Identifying and Selecting Evidence-Based Interventions

and conditions specified in the community logic model. Below is a list of utility and feasibility
checks to consider in selecting prevention strategies.

Utility and Feasibility Checks

Utility Checks

• Isassessment
the intervention appropriate for the population identified in the community needs
and community logic model? Has the intervention been implemented suc-
cessfully with the same or a similar population? Are the population differences likely to
compromise the results?

• Iswhat
the intervention delivered in a setting similar to the one planned by the community? In
ways is the context different? Are the differences likely to compromise the interven-
tion’s effectiveness?

• Isparticipate
the intervention culturally appropriate? Did members of the culturally identified group
in developing it? Were intervention materials adapted to the culturally identi-
fied group?

• Are implementation materials (e.g., manuals, procedures) available to guide intervention

implementation? Are training and technical assistance available to support implementa-
tion? Are monitoring or evaluation tools available to help track implementation quality?

Feasibility Checks

• Is the intervention culturally feasible, given the values of the community?

• Isimplementing
the intervention politically feasible, given the local power structure and priorities of the
organization? Does the intervention match the mission, vision, and culture
of the implementing organization?

• Isimplementing
the intervention administratively feasible, given the policies and procedures of the
organization?

• Isprogram
the intervention technically feasible, given staff capabilities and time commitments and
resources?

• Is(including
the intervention financially feasible, given the estimated costs of implementation
costs for purchase of implementation materials and specialized training or
technical assistance)?

Each of the points in the checklist warrants thoughtful consideration among those involved in
planning, implementing, and evaluating the prevention strategies in the comprehensive commu-
nity plan.

Identifying and Selecting Evidence-Based Interventions 11

 IV. Using Public Resources/Review Processes to
Identify Evidence-Based Interventions and
Determine Their Evidence Status
Evidence-Based Interventions and Evidence Status
Experts in the field agree that the nature of evidence is continuous. The strength of evidence or
“evidence status” of tested interventions will fall somewhere along a continuum from weak to
strong. Strength of evidence is traditionally assessed using established scientific standards and
criteria for applying these standards. Strength of evidence comprises three major elements:

• Rigor of the study design (e.g., use of appropriate comparison and control groups; time
series design).

• Rigor and appropriateness of the methods used to collect and analyze the data
(e.g., whether data were collected in an unbiased manner and the statistical tests
were appropriate).

These two elements directly affect the inferences that can be drawn about cause and
effect—the degree to which the results obtained from an evaluation can be attributed
to the intervention exclusively, rather than to other factors.

• The extent to which findings can be generalized to similar populations and settings.
This element refers to the likelihood that the same findings will be obtained if the inter-
vention is repeated in similar circumstances.

Strong evidence means that the intervention “works”—that it generates a pattern of positive
outcomes attributed to the intervention itself, and that it reliably produces the same pattern of
positive outcomes for certain populations under certain conditions.

Experts agree that evidence becomes “stronger” with replication and field testing in various cir-
cumstances. However, experts do not agree on a specific minimum threshold of evidence or cutoff
point below which evidence should be considered insufficient. Nor do they agree whether little
evidence is equivalent to no evidence at all. Even evidence from multiple studies may still be
judged insufficient to resolve all doubts about the likely effectiveness of an intervention designed
for a different population or situation.

This discussion takes us to the role of professional judgment and the application of critical think-
ing skills to determine overall best fit of interventions to include in a comprehensive community
plan. Strength of evidence is critical to selecting interventions that are likely to work, but it is not
the sole consideration. Keep in mind two practical criteria:

12 Identifying and Selecting Evidence-Based Interventions

 1. Out of two interventions, choose the one for which there is stronger evidence of effec-
tiveness, if the intervention is similar, equivalent, and equally well-matched to the com-
munity’s unique circumstances.

2. Reserve selecting an intervention with little or weak evidence of effectiveness for situa-
tions in which other interventions with stronger evidence do not fit local circumstances.

SPF Definitions of Evidence-Based Status

The SPF SIG Program specifically requires implementation of evidence-based interventions.
Evidence-based interventions are defined in the SPF SIG Program by inclusion under one or more
of three public resources/review mechanisms that rate, make judgments, or provide information
about the strength of evidence supporting specific interventions. These definitions or resource
mechanisms are as follows:

• Included on Federal Lists or Registries of evidence-based interventions;

• Reported (with positive effects) in peer-reviewed journals; or
• Documented effectiveness based on the three new guidelines for evidence.
Each of the three definitions helps identify evidence-based interventions and each presents its
own advantages and challenges.

Regardless of the resource or review process, consumers must be prepared to think critically
about the adequacy of evidence for interventions deemed relevant (conceptual fit) in the logic
model and appropriate (practical fit) for real-world implementation.

A. Using Federal Lists or Registries

Federal Lists or Federal Registries are readily accessible and easy-to-use public resources.
Historically, most Federal Lists or Registries are limited in scope since they are geared to interven-
tions most amenable to assessment using traditional research designs and methodologies for
evaluation. These interventions typically share certain characteristics:

• Discrete in scope;
• Guided by curricula or manuals;
• Implemented in defined settings or organized contexts; and
• Focused primarily on individuals, families, or defined settings.

Identifying and Selecting Evidence-Based Interventions 13

 Advantages

Federal Lists and Registries—

• Provide concise descriptions of discrete interventions;

• Provide documented ratings of strength of evidence measured against defined and gen-
erally accepted standards for scientific research;

• Present a variety of practical information, formatted and categorized for easy access, and
potentially useful to implementers; and

• Offer “one-stop” convenience for those seeking quick information on certain types of
interventions.

Challenges

Federal Lists and Registries—

• Include a limited number of interventions. Not all those eligible choose to apply. Also, the
availability of funding may limit the number of interventions that can be reviewed and
included in a Registry at any given time;

• Include the types of interventions most easily evaluated using traditional scientific stan-
dards and research methodologies. Historically, this has resulted in an overrepresentation
of school-based and individual-focused interventions and an underrepresentation of
environmental and community-based interventions;

• Use review criteria that emphasize the importance of internal validity (attribution of
results to the intervention only) over external validity (ability to generalize to other popu-
lations, contexts, and real-world situations); and

• Confer misleading “global effectiveness labels” based on arbitrary cutoff points along an
evidence continuum (sometimes with minuscule differences between those included
in a particular category and those excluded) and often overgeneralize outcomes not
measured in the study.

SAMHSA’s National Registry of Evidence-Based Programs and Practices (NREPP) is a decision sup-
port system designed to help stakeholders (including States and community-based organiza-
tions) select interventions. The NREPP reflects current thinking that States and communities are
best positioned to decide what is most appropriate for their needs.

Scheduled to be up and running early in calendar year 2007, SAMHSA’s new NREPP will be avail-
able to local prevention providers and decision makers seeking to identify interventions that pro-
duce specific community outcomes. Reconceptualized as a decision-support tool, the new NREPP

14 Identifying and Selecting Evidence-Based Interventions

represents a significant policy accommodation by SAMHSA on behalf of decision makers needing
a more diverse set of options to address broader community problems.

Key points about NREPP are as follows:

• NREPP is a voluntary rating and classification system designed to provide the public with
reliable information on the scientific basis and practicality of interventions that prevent
and/or treat mental and substance use disorders.

• Outside experts will review and rate interventions on two dimensions: strength of evi-
dence and dissemination capability. Strength of evidence is defined and assessed on six
criteria; readiness for dissemination is defined and assessed on three criteria. Each criteri-
on will be numerically rated on an ordinal scale ranging from zero to four.

• For all interventions reviewed, detailed descriptive information and the overall average
rating score on each dimension (regardless of the rating score) will be included and post-
ed on the NREPP Web site. Average scores achieved on each rating criterion within each
dimension will also be available on the NREPP Web site (www.nrepp.samhsa.gov).

• NREPP allows a broader range of evaluation research designs to be eligible for review,
including single group pre/posttest design without comparison or control data. However,
to encourage the submission of interventions likely to receive strong reviews (i.e., those
that demonstrate strength of evidence), NREPP establishes three minimum or threshold
requirements that must be met:

1. The intervention demonstrates one or more positive changes (outcomes) in mental

health and/or substance use behavior among individuals, communities, or populations;

2. Intervention results have been published in a peer-reviewed publication or documented

in a comprehensive evaluation report; and

3. Documentation (e.g., manuals, process guides, tools, training materials) of the interven-
tion and its proper implementation is available to the public to facilitate dissemination.

In addition to the threshold evidence requirements, NREPP will award “priority review points” for
quality of study design and for outcomes in designated content areas. Priority points increase
the potential for qualifying applications to be selected for review. Interventions will receive one
priority point if they have been evaluated using a quasi-experimental or experimental study
design, including a pre/post design with comparison or control group, or longitudinal/time series
design with a minimum of three data points, one of which must be a baseline assessment.

Identifying and Selecting Evidence-Based Interventions 15

 B. Using Peer-Reviewed Journals
Peer-reviewed journals present findings about what works and what does not. The burden for
determining the applicability and credibility of the findings falls on the reader.

Advantages

Peer-reviewed journals—

• Preview new and emerging prevention strategies; highlight a program, practice, or

local policy initiative for further follow-up directly with the intervention developer/
implementer;

• Report and summarize meta-analyses and other types of complex analyses (e.g., core
components) that examine effectiveness across interventions or intervention compo-
nents; and

• Present detailed findings and analyses that illuminate whether or not and how an inter-
vention works.

Challenges

Peer-reviewed journals—

• Leave it to the reader to assess the credibility of evidence presented and its relevance
and applicability to the community;

• Describe in limited detail the activities and implementation issues pertinent to dissemi-
nation; and

• Emphasize the importance of internal validity (attribution of results to the intervention)

over external validity (generalizability to different populations and contexts).

Assessing Elements of Evidence Reported in Peer-Reviewed Journals

Using the primary research literature to identify potential prevention interventions requires criti-
cal assessment of the quality of the research presented and the conceptual model on which it is
based. Listed below are key elements addressed in most peer-reviewed journal articles along
with some question probes. Critical consumers of information presented in peer-reviewed jour-
nals should be prepared to read each article at least twice.

• Background on the intervention evaluated in the study. Does the article adequately set the
stage for the study and describe why the study was undertaken? Does it adequately
describe the intervention? The characteristics of the populations involved in the study?
The context or setting of the intervention? How closely does the objective of the study
reflect the needs of your community?

16 Identifying and Selecting Evidence-Based Interventions

 • Atermdefined conceptual model that includes definitions and measures of intermediate and long-
outcomes. Does the article describe the theory base of the intervention and link the
theory to expectations about the way the program works and specific outcomes expect-
ed? Does the article describe the connection of theory to intervention approach and
activities, and to expected outcomes, in sufficient detail to guide your implementation?

• Apopulation
well-described study population that includes baseline or “pre” measurement of the study
and comparison or control groups included in the study. Does the article
describe the characteristics of the study population and comparison/control groups?
How well does the study population match your local target group? How are they similar
or different?

• Overall quality of study design and data collection methods. Does the overall study design
adequately rule out competing explanations for the findings? Did the data collection
methods account for participant attrition? Missing data? Data collector bias and selection
bias? Did the study methodology use a combination of strategies to measure the same
outcome using different sources (converging evidence)? Is the overall study design suffi-
ciently robust to show that the intervention worked?

• Analytic plan and presentation of the findings. Does the analytic plan address the questions
posed in the study? Does the article report and clearly describe findings/outcomes and
do they track with what was expected?

• Aclusions
summary and discussion of the findings. Does the discussion draw inferences and con-
that are appropriate and grounded in the findings and strength of the overall
study design?

C. Using Guidelines for Documented Evidence of Effectiveness

Some complex interventions, which usually include innovations developed locally, look different
from most of those in Federal Lists and Registries. Because complex interventions exhibit qualities
different from those of discrete and manualized interventions, they may require customized
assessment. Complex interventions may exhibit certain characteristics that make them difficult to
evaluate and measure:

• A multifaceted approach with interacting components;

• Inclusive outreach across populations and settings—targeting heterogeneous groups
of participants, spanning a range of settings, and extending across multiple levels of
organization;

• Aopportunities;
philosophy that values adaptation in response to unique community needs and

Identifying and Selecting Evidence-Based Interventions 17

 • Reliance on the involvement of committed individuals who provide informal services that
go beyond those planned; and

• Acommunity
flexible intervention design that responds readily to unpredictable and changing
circumstances.

SPF SIG Program Guidelines for Documented Effectiveness

The SAMHSA/CSAP Expert Workgroup recommended taking a broad view toward judging the
adequacy of evidence for complex interventions. It recommended using different types or
streams of evidence, drawing from traditional research-designed evaluation studies as well as
accumulated local empirical data, established theory, professional experience, and indigenous
local knowledge and practitioner experience.

Central to the Expert Workgroup’s recommendations is the concept of blending—combining

multiple streams of evidence to support an optimal mix of interventions to include in a compre-
hensive community plan.

The Expert Workgroup recognized that evidence provided as support for community-based inter-
ventions must reflect certain characteristics to be credible and persuasive. These characteristics
are captured in three guidelines for evidence all of which must be met to demonstrate “documented
effectiveness” under the SPF SIG Program:

Guideline 1: The intervention is based on a solid theory or theoretical perspective that has
been validated by research;

Guideline 2: The intervention is supported by a documented body of knowledge—a con-

verging accumulation of empirical evidence of effectiveness—generated from similar or
related interventions that indicate effectiveness; and

Guideline 3: The intervention is judged by a consensus among informed experts to be

effective based on a combination of theory, research, and practice experience. Informed
experts may include key community prevention leaders, and elders or other respected
leaders within indigenous cultures.

These guidelines are intended to expand the array of interventions available to prevention planners;
they are considered supplements, not replacements, for traditional scientific standards in Federal
evidence-rating systems or peer-reviewed journals.

Communities are encouraged to use as many types of documentation as possible to justify

selecting a particular complex, evidence-based intervention.

Notice that these guidelines do not specify a minimum threshold level of evidence of effectiveness.
They rely instead on professional judgment to determine the adequacy of evidence to meet these
three guidelines when considered in the broader context of the comprehensive community plan.

18 Identifying and Selecting Evidence-Based Interventions

 Advantages

Guidelines for documented evidence of effectiveness—

• Enable State and community planners to diversify the portfolio of strategies incorporated
in a comprehensive plan; ensure flexibility for those making programming decisions;

• Empower State and community planners to select or develop innovative, complex inter-
ventions to meet the needs of individual communities;

• Create the potential for using culturally based evidence as well as traditional evidence to
support local decisions; and

• Authorize State and community planners to exercise professional judgment in deciding

the potential contribution of unique intervention components in the comprehensive plan.

Challenges

Guidelines for documented evidence of effectiveness—

• Place substantial responsibility on prevention planners for intervention selection deci-

sions. The guidelines are new and are neither simple nor simplistic; and

• Require prevention planners to think critically about the evidence provided to support
the inclusion of a particular intervention in the community’s comprehensive plan.

Examples of Evidence to Support Documented Effectiveness

Several types of evidence may be used to support documented effectiveness as defined under
the SPF SIG Program. Documentation is important to justify the inclusion of a particular interven-
tion in a comprehensive community plan. Prevention planners are encouraged to provide as
many types of documentation as are appropriate and feasible in order to provide strong justifica-
tion of documented effectiveness. The following are types of documented evidence that may be
used to demonstrate documented effectiveness:

• Documentation that clarifies and explains how the intervention is similar in theory, content,
and structure to interventions that are considered evidence-based by scientific standards.

• Documentation that the intervention has been used by the community through multiple
iterations, and data collected indicating its effectiveness.

• Documentation that indicates how the proposed intervention adequately addresses ele-
ments of evidence usually addressed in peer-reviewed journal articles. These elements
may include the nature and quality of the evaluation research design; the consistency of
findings across multiple studies; and the nature and quality of the data collection meth-
ods, including attention to missing data and possible sources of bias.

Identifying and Selecting Evidence-Based Interventions 19

 • Documentation that explains how the proposed intervention is based on an established
theory that has been tested and empirically supported in multiple studies. This docu-
mentation should include an intervention-specific logic model that details how the pro-
posed intervention applies and incorporates the established theory.

• Documentation that explains how the proposed intervention is based on published prin-
ciples of prevention. This documentation should provide references for the principles
cited and should explain how the proposed intervention incorporates and applies these
principles.

• Documentation that describes and explains how the intervention is rooted in the indige-
nous culture and tradition.

20 Identifying and Selecting Evidence-Based Interventions

V. Summary Process Description: Selecting Best Fit
Prevention Interventions
The process described here is rooted in the work conducted by local communities during SPF
Steps 1 and 2. It begins with a community logic model to map the local substance abuse picture
and draws from the findings of local needs and resource assessment. Prevention planners apply
the logic model and assessment findings in a process of thinking critically and systematically
about three considerations that determine best fit interventions to include in a comprehensive
plan:

• Conceptual fit with the community’s logic model (is it relevant?);

• Practical fit with the community’s needs, resources, and readiness to act
(is it appropriate?); and

• Evidence of effectiveness (is it effective?).

Figure 3 depicts the process for thinking through these key considerations.

Figure 3. Process Description: Selecting Best Fit Prevention Interventions

Identify types of
interventions that Select specific programs,
practices, and policies that AND
• address a community’s
• are adequately
salient risk and Best fit
• are feasible given a community’s supported by theory,
protective factors prevention interventions
resources, capacities, and empirical data, and the
and contributing to include in
readiness to act consensus judgment
conditions comprehensive
of informed experts
• add to/reinforce other strategies and community community plan
• target opportunities for
in the community–synergistic vs. prevention leaders
intervention in multiple
duplicative or stand-alone efforts
life domains

• drive positive outcomes

in one or more substance Demonstrate
abuse problems, “Evidence of
consumption patterns, or Effectiveness”
consequences Demonstrate Effective?
“Practical Fit”

Appropriate?
Demonstrate
“Conceptual Fit”

Relevant?

Identifying and Selecting Evidence-Based Interventions 21

 VI. SPF SIG Program Guidance: Roles and Expectations
Collaboration and partnership across all levels—Federal, State, and community or local grantee—
are essential for successful and flexible implementation of the guidance in this document. The
guidance details an analytic process and a few key concepts—what needs to be done to think
through the selection of best fit evidence-based prevention interventions. How this is accom-
plished will be determined by States and jurisdictions and will vary from one to another.
SAMHSA/CSAP’s technical assistance providers are available to work with States and jurisdictions
to apply the process and concepts detailed in the guidance.

A. Federal Role
SAMHSA/CSAP will provide leadership and technical assistance to States and jurisdictions and will
work with them to strengthen prevention systems in order to improve substance use outcomes
and achieve targeted community change.

Expectations

• SAMHSA/CSAP will partner with States to develop and implement a plan that facilitates
application of the guidance.

• SAMHSA/CSAP, with its technical assistance providers, will work with States to develop
their system capacities to support communities in selecting interventions. To this end,
SAMHSA/CSAP has directed its five regional Centers for the Application of Prevention
Technologies (CAPTs) to allocate substantial technical assistance resources for States to
apply the concepts in this guidance. At the request of States, CAPTs will conduct work-
shops and activities to help States work with communities to identify and select suitable
and effective evidence-based interventions.

B. State/Jurisdiction Role
The role of the States and jurisdictions is to provide capacity-building activities, tools, and
resources to communities to foster the development of sound community prevention systems
and prevention strategies.

Expectations

• SAMHSA/CSAP expects States funded under the SPF SIG Program to strengthen their infra-
structure and capacity to assist communities in identifying and selecting evidence-based
interventions for their comprehensive plans.To accomplish this, SAMHSA/CSAP expects
States to establish a mechanism (e.g., technical expert panel) to assure accountability for:
reviewing comprehensive community plans and the justification for interventions included
in the plan; identifying issues and problematic intervention selections; and targeting techni-
cal assistance to work with communities to improve and strengthen their community plans.

22 Identifying and Selecting Evidence-Based Interventions

 In thinking about the implications of this guidance, States may want to consider the
questions below:

How might your State engage informed experts, including community leaders, in
applying the concepts in the guidance for funding comprehensive community plans
(programs, practices, and policies) selected by your communities?

How might your State communicate its policies regarding funding and implementa-
tion of evidence-based programs, practices, and policies to community coalitions
and organizations and other key stakeholders?

• SAMHSA/CSAP expects States, with their technical assistance providers, to work closely with
communities in identifying and selecting evidence-based interventions. SAMHSA/CSAP and
its technical assistance providers will work directly with States on this task.

• SAMHSA/CSAP expects States to develop capacities to assist communities on all key SPF
topics, including assessing needs and resources; using data to detail the substance abuse
problem and underlying factors and conditions; building a community logic model; and
examining intervention options for relevance and appropriateness.

C. Community Role
The role of SPF SIG subrecipient communities is to develop a comprehensive and strategic commu-
nity prevention plan based on local needs and resource assessment. Following the steps of the SPF,
communities use the findings from these activities to develop a logic model specific to the commu-
nity and its substance abuse problem. Each community logic model reflects and maps the local sub-
stance abuse phenomenon. An effective logic model may serve as the primary tool to guide the
selection of evidence-based programs, practices, and policies to include in a comprehensive plan.

Expectations

• SAMHSA/CSAP expects communities to partner with the State and its technical assis-
tance providers, who in turn will partner with SAMHSA/CSAP and CSAP’s technical
assistance providers.

Concluding Comments
As in all steps of SAMHSA’s Strategic Prevention Framework, the application of critical thinking skills
is vital to selecting programs, practices, and policies to include in a comprehensive strategic plan.
Those selected must be relevant, appropriate, and effective to meet community needs and address
the community substance abuse problem. SAMHSA/CSAP and its technical assistance providers wel-
come the opportunity to partner with SPF SIG States, jurisdictions, and federally recognized tribes
and tribal organizations through technical assistance workshops and “science to service” learning
communities to think through the selection of best fit evidence-based prevention interventions.

Identifying and Selecting Evidence-Based Interventions 23

 GLOSSARY
Best fit interventions Interventions that are relevant to the community logic model
(i.e., directed to the risk and protective factors most at play in a
community) and appropriate to the community’s needs,
resources, and readiness to act.

Community logic model A graphic depiction or map of the relationships between the
local substance abuse problem, the risk/protective factors and
conditions that contribute to it, and the interventions known to
be effective in altering those underlying factors and conditions.

Conceptual fit The degree to which an intervention targets the risk and protective
factors that contribute to or influence the identified community
substance abuse problem.

Documented effectiveness Defined under the SPF SIG Program by guidelines for evidence
to demonstrate intervention effectiveness. These guidelines
include grounding in solid theory, a positive empirical track
record, and the consensus judgment of informed experts and
community prevention leaders.

Epidemiological profile A summary and characterization of the consumption (use) pat-

terns and consequences of the abuse of alcohol, tobacco, mari-
juana, heroin, cocaine, methamphetamines, inhalants, prescrip-
tion drugs, or other substances. The epidemiological profile
identifies the sources of data on consumption patterns as well
as the indicators used to identify consequences (e.g., morbidity
and mortality). It should provide a concise, clear picture of the
burden of substance abuse in the State using tables, graphs, and
words as appropriate to communicate this burden to a wide
range of stakeholders.

Evidence-based interventions Interventions based on a strong theory or conceptual framework

that comprise activities grounded in that theory or framework
and that produce empirically verifiable positive outcomes when
well implemented.

Evidence-based status— Defined by inclusion through one or more of three public

SPF SIG program resources or review processes that make judgments and
provide information about the strength of evidence for
intervention selections:

24 Identifying and Selecting Evidence-Based Interventions

 • Included on Federal Lists or Registries of evidence-based
interventions;
• Reported (with positive outcomes) in peer-reviewed
journals; or
• Documented effectiveness based on guidelines developed
by SAMHSA/CSAP.

Evidence status or strength Refers to the continuum of evidence quality which ranges from
of evidence weak to strong. Strong evidence means that the positive outcomes
assessed are attributable to the intervention rather than extrane-
ous events and that the intervention reliably produces the same
pattern of positive outcomes in similar populations and contexts.
Strong evidence means that the intervention works.

External validity The extent to which evaluation outcomes will be achieved in

populations, settings, and timeframes beyond those involved in
the study; the likelihood that the same pattern of outcomes will
be obtained when the intervention is implemented with similar
populations and in similar contexts.

Internal validity The extent to which the reported outcomes can be unambigu-
ously attributed to the intervention rather than to other com-
peting events or extraneous factors.

Interventions Interventions encompass programs, practices, policies, and

strategies that affect individuals, groups of individuals, or entire
communities.

Outcomes-based prevention An approach to prevention planning that begins with a solid

understanding of a substance abuse problem, progresses to
identify and analyze factors/conditions that contribute to the
problem, and finally matches intervention approaches to these
factors/conditions ultimately leading to changes in the identified
problem, i.e., behavioral outcomes.

Practical fit The degree to which an intervention meets the resources and
capacities of the community and coincides with or matches the
community’s readiness to take action.

Protective factors Conditions for an individual, group, or community that decrease

the likelihood of substance abuse problems and buffer the risks
of substance abuse.

Risk factors Conditions for an individual, group, or community that increase

the likelihood of a substance abuse problem.

Identifying and Selecting Evidence-Based Interventions 25

SMA-4205