The Scientific Status of Psychotherapies: A New Evaluative

Framework for Evidence-Based Psychosocial Interventions

Daniel David, Department of Oncological Sciences, Mount Sinai School of Medicine and
Department of Clinical Psychology and Psychotherapy, Babeş-Bolyai University
Guy H. Montgomery, Department of Oncological Sciences, Mount Sinai School of Medicine

The meaning of the term evidence-based psychotherapy PROBLEMS WITH THE PRESENT EVIDENCE-BASED

(EBP) is a moving target and is inconsistent among PSYCHOTHERAPIES CLASSIFICATION SYSTEMS

international organizations. To clarify the meaning of The evidence-based movement within the psychologi-
EBP and to provide guidelines for evaluating psychoso- cal community strives to improve the efficacy of psy-
cial interventions (i.e., psychological treatments), we
chosocial ⁄ psychological interventions ⁄ treatments (i.e.,
psychotherapies) as a whole, as well as to provide
propose that psychotherapies should be first classified
treatment guidelines for clients, professional providers,
into nine categories, defined by two factors: (a) theory
and third-party payers alike. Recently, we have wit-
(mechanisms of psychological change) and (b) therapeu-
nessed a proliferation of evaluative frameworks for evi-
tic package derived from that theory, each factor orga-
dence-based psychotherapies ⁄ psychological treatments
nized by three levels: (a) empirically well supported; (b) (i.e., empirically validated therapies, empirically sup-
equivocal data [(a) no, (b) preliminary data less than ported therapies). However, there is a problem associ-
minimum standards, or (c) mixed data]; and (c) strong ated with having multiple evaluative systems within
contradictory evidence. As compared to the previous the field. Specifically, multiple evaluative frameworks
classification systems, and building on them, we add for evidence-based psychotherapies have led to con-
the requirement that there should also be a clear rela- flicting views and standards regarding the status of
tionship between a guiding theoretical base and the individual psychological interventions. That is, psycho-
empirical data collected. The proposed categories are logical treatments may be labeled ‘‘evidence-based’’ in
not static systems; depending on the progress of
one system, but not in others. For example, the
National Institute for Health and Clinical Excellence
research, a form of psychotherapy could move from
Guidelines (NICE’s Guidelines; http://www.nice
one category to another.
.org.uk) are not always consistent with those of the
Key words: classification framework of psychosocial/
American Psychological Association (APA ⁄ Division
psychological interventions/treatments, evidence-based
12 ⁄ Society for Science of Clinical Psychology ⁄
psychotherapies. [Clin Psychol Sci Prac 18: 89–99, SSCP; http://www.psychology.sunysb.edu/eklonsky-/
2011] division12/), with those of the American Psychiatric
Association (http://www.psych.org), or with what
we learn from Cochrane Reviews (http://www
Address correspondence to Professor Daniel David, Ph.D., .cochrane.org). This lack of consistency generates con-
Head of Department of Clinical Psychology and Psychotherapy, fusion among professionals and patients alike who are
Babes-Bolyai University, No. 37 Republicii St., 400015 looking to use empirically validated treatments and
Cluj-Napoca, Romania. E-mail: [email protected]. strongly supports the need of a unified, more

 2011 American Psychological Association. Published by Wiley Periodicals, Inc., on behalf of the American Psychological Association.
All rights reserved. For permissions, please email: [email protected] 89
complex, and scientifically oriented system for catego- suggesting that malaria is caused by a pathogen dissemi-
rizing psychological treatments. nated by the anopheles mosquito, interventions to fight
Furthermore, all the current systems of evaluating malaria were dramatically improved. Based on these
evidence-based psychotherapies have a significant well-known examples, it is easy to imagine how the
weakness; they restrict their focus on evidence to data inclusion of ‘‘voodoo’’- and ⁄ or ‘‘bad air’’–based theory
supporting (psycho)therapeutic packages while ignoring interventions could be damaging to the entire field of
whether any evidence exists to support the proposed health-related interventions.
theoretical underpinnings of these techniques (i.e., the- Therefore, to promote the field of psychotherapy,
ory about psychological mechanisms of change; see from both scientific and clinical perspectives, we propose
David, 2004).1 Therefore, by ignoring the theory, the a new evaluative framework for categorizing psychologi-
evaluative frameworks of various health-related inter- cal interventions. We hope that this framework can lead
ventions (including psychotherapy), technically (a) to increased uniformity in evidence-based psychothera-
allow pseudoscientific (i.e., ‘‘junk-science’’) interven- pies evaluation guidelines and also separate scientific
tions to enter into the classification schemes and ⁄ or (b) approaches to psychotherapy from pseudoscientific ones.
bias the scientific research in a dangerous direction. For Following similar articles in the field (e.g., Chambless
example, imagine a hypothetical intervention to man- et al., 1998), we have decided not to discuss here specific
age psychological symptoms that is based on ‘‘voodoo’’ and detailed examples of psychotherapies in each cate-
as its underlying theory about the mechanisms of gory, as they would all require a detailed analysis based
change. Imagine this therapeutic package being sup- on the criteria in each category. Moreover, positioning
ported by randomized trial data (e.g., better than wait- into a category would depend on the disorder to which a
ing list [BWL]). Such an intervention could then be specific treatment is applied. Such an analysis would be
classified as a ‘‘probably efficacious treatment’’ accord- too long for the scope of the current article and will be
ing to current evaluation guidelines (see Chambless conducted in an independent paper. However, we dis-
et al., 1996, 1998) despite the therapeutic package cuss here ‘‘very strong’’ and ‘‘very weak’’ treatments to
being based on a theory (‘‘voodoo’’) that at best is clearly illustrate the proposed system.
highly questionable. Closer to our field, a similar analy-
sis has been conducted by McNally (1999), historically A NEW EVALUATIVE FRAMEWORK FOR EVIDENCE-BASED
comparing eye movement desensitization and reproces- PSYCHOTHERAPIES
sing (EMDR) and mesmerism. That is, a consequence We propose an evaluative, hierarchical framework for
of current classification schemes (which consistently do psychotherapy, which is based on the understanding
not address underlying theories about mechanisms of that there are two levels in the analysis of evidence
change) is that as long as there are randomized trial supporting psychological treatments. First, the psycho-
data, the validity of the underlying theory is less rele- logical theory concerning therapeutic change (e.g.,
vant. As concerning the issue of negatively biasing the mechanisms of change) should be scientifically evalu-
research field, not that long ago it was commonly ated. Second, the therapeutic package (psychological
believed that malaria was produced by ‘‘bad air’’ (hence treatment) is derived from the theory about the
the name). Based on the ‘‘bad air’’ theory, an effective mechanisms of change and is scientifically evaluated
intervention was developed: closing room windows to (David, 2004). Interventions (e.g., acupuncture) that
prevent the circulation of bad air. In light of what we do not explicitly have a psychological basis (theory
know about malaria now, it is not surprising that this and techniques) are typically excluded from this
intervention was partially effective. If scientists were analysis, although they may work in part by psycho-
satisfied with the ‘‘bad air’’ theory and its ‘‘effective- logical mechanisms (e.g., expectancies); however, they
ness,’’ we might still be attempting to develop better can be analyzed through this classification scheme, as
windows to better control malaria. Once the flaws in a more general part of health-related interventions, if
the ‘‘bad air’’ theory were recognized (in spite of its they target psychological and ⁄ or psychosomatic symp-
partial ‘‘effectiveness’’) and were replaced by a theory toms (see above the analysis of ‘‘voodoo’’). Indeed,

CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE • V18 N2, JUNE 2011 90

the proposed scheme can also be applied to other
Table 1. Psychotherapies Classification Framework: Categories I–IX
psychosocial interventions, which are not necessarily
therapeutic.
Psychotherapies should be classified into nine cate-
gories, defined by two factors (see Table 1): (a) theory
(i.e., about psychological mechanisms of change) and
(b) therapeutic package derived from that theory, each
factor organized by three levels: (a) empirically well
supported; (b) equivocal ⁄ no clear data [(a) not yet
been collected, (b) preliminary data (PD) less than
minimum standards, or (c) mixed (both supporting
and contradictory evidence) data]; and (c) strong con-
tradictory evidence (SCE; i.e., invalidating evidence). Notes. aWell-supported theories are defined as those with evidence based
on (a) experimental studies (and sometimes additional ⁄ adjunctive correla-
By supporting evidence we mean evidence of benefit tional studies) and ⁄ or (b) component analyses, patient · treatment inter-
actions, and ⁄ or mediation ⁄ moderation analyses in complex clinical trials
(beneficence). By contradictory (invalidating) evidence (CCTs); thus, the theory can be tested independent of its therapeutic
we mean evidence of absence of benefit (inert) package (e.g., in experimental studies and sometimes their additional ⁄
adjunctive correlational studies) and ⁄ or during a CCT; ‘‘well supported’’
and ⁄ or evidence of harm (malfeasance). The proposed within this framework means that it has been empirically supported in at
categories are not static systems; depending on the least two rigorous studies, by two different investigators or investigating
teams.
b
progress of research, a form of psychotherapy could Equivocal evidence for therapeutic package and ⁄ or theory means No
(data not yet collected), Preliminary (there is collected data, be they
move from one category to another. Also, the pro- supporting or contradictory, but they do not fit the minimum standards),
posed categories can separate ‘‘scientifically’’ from or Mixed Data (MD; there is both supporting and contradictory evi-
dence).
‘‘pseudoscientifically’’ oriented psychotherapies, with c
Strong contradictory evidence (SCE) for therapeutic package and ⁄ or the-
ory means that it has been empirically invalidated in at least two rigorous
major theoretical (e.g., what to teach and research) studies, by two different investigators or investigating teams.
d
and practical implications (e.g., what to recommend Well-supported therapeutic packages are defined as those with random-
ized clinical trial (or equivalent) evidence of their efficacy (absolute, rela-
as good scientific practices). tive, and ⁄ or specific) and ⁄ or effectiveness; ‘‘well supported’’ within this
Scientifically oriented psychotherapies (SOPs) are framework means that it has been empirically supported in at least two
rigorous studies, by two different investigators or investigating teams.
those which do not have clear SCE for theory and • Red signifies pseudoscientifically oriented psychotherapies (POPs); the
core of POPs (darker red) is represented by Category IX. Green signifies
package; the highest level of validation of a SOP is that scientifically oriented psychotherapies (SOPs); the core of SOPs (darker
in which both the theory about psychological mecha- green) is represented by Category I.
• Depending on the progress of research, a psychotherapy could move
nisms of change and the therapeutic package are well from one category to another.
• Example of Coding. A psychotherapy, X, from Category I, may be
validated (i.e., Category I). A SOP seeks to investigate analyzed in details (i.e., within category analysis)—if necessary and rele-
empirically both the therapeutic package in question vant —by coding it according to the codes described in the article. The
order of coding is Category ⁄ Theory (with nuances separated by ‘‘;’’ ⁄
and the underlying theory guiding the design and Therapeutic package (with nuances indicated and separated by ‘‘-‘‘ ‘‘:’’
implementation of the therapeutic package (i.e., theory ‘‘,’’ ‘‘;’’); the numeric codes indicate the number of studies. For exam-
ple, if psychotherapy X is coded (this is a complex example) ‘‘I ⁄ 2,I,
about mechanisms of change). In this way, the ITT,E;2,I,CTT ⁄ 3CTAE:BWL;RE:BST,BC:BWL;SE:BST,MM’’ this means
that (narrative description):
proposed framework rules out the inclusion of ‘‘voo- • it belongs to Category I;
doo’’-like psychotherapy and prevents us from devel- • its theory has been empirically supported in at least two rigorous
studies, by two different investigators or investigating teams (I); the
oping ‘‘bad air’’–like theories. Theory refers to the theory has been tested both independent of its therapeutic package, in
two (2) experimental (E) studies, (ITT), and in two (2) complex clinical
mechanisms of change, namely the hypothesized psy- trials;
chological factors involved in pathology and health, • its absolute efficacy shows (in three studies—clinical trials, CT) that it is
better than a wait-list control condition (BWL);
which are targeted by the therapeutic package. Indeed, • its relative efficacy (RE) shows that it is better than another evidence-
there should be a correspondence between the mecha- based psychological intervention (BST) and both are better than control
conditions (BC), in the form of waiting list (BWL);
nisms of treatment (‘‘mechanism ⁄ theory of change’’) • its specific efficacy (SE) shows that it is significantly better than other
active ⁄ standard therapies (BST) and the underlying theory is based on
and the mechanisms of the disorder (‘‘theory of disor- analyses of mediation and ⁄ or moderation (MM).
der’’). A specific treatment (and its mechanisms of s Being a complex approach, the coding profile should be always
accompanied by a narrative description, as presented above.
change) is more scientifically legitimate if it is derived

NEW EVALUATIVE FRAMEWORK FOR EVIDENCE-BASED PSYCHOTHERAPIES • DAVID & MONTGOMERY 91

from experimental psychopathology research that has which are targeted by the therapeutic package. Of
clarified the nature of the disorder. course, a validated theory is not an ‘‘all or nothing’’
Pseudoscientifically oriented psychotherapies (POPs) decision. Rather, it is a continuing and developing pro-
are those that claim to be scientific, or that are made to cess as the scientific evidence accumulates from various
appear scientific, but that do not adhere to an appropri- study designs (e.g., correlational, experimental) and
ate scientific methodology (e.g., there is an overreliance types (e.g., clinical, analogue). We will not provide an
on anecdotal evidence and testimonial rather than epistemological discussion of what a ‘‘validated’’ (sup-
empirical evidence collected in controlled studies; ported) theory or therapeutic package is; we will just
Lilienfeld, Lynn, & Lohr, 2003). The term pseudoscience mention that validating a theory refers to testing it,
cannot be rigorously categorically defined; a prototypic based on a current scientific approach (e.g., falsifiability,
definition, based on a number of themes, is used verifiability).
instead (see Lilienfeld et al., 2003). We define POPs as Consistent with published criteria for treatments
therapies used and promoted in the clinical field as if (Chambless et al., 1998), we argue that a theory is well
they were scientifically based, despite strong contrary supported, within this framework, if it has been empir-
evidence related to at least one of their two compo- ically validated in at least two rigorous studies, by two
nents (i.e., therapeutic package and theory). Indeed, a different investigators or investigating teams (I). The
specific treatment may sometimes not be related to theory can be tested (a) independent of its therapeutic
‘‘pseudoscience-like themes,’’ but it is the nature package (independent theory testing [ITT]; e.g., in
and ⁄ or the degree of its promotion that outstrips the experimental (E) and sometimes additional ⁄ adjunctive
available evidence that may qualify it as pseudoscientific correlational (C) studies) and ⁄ or (b) during complex
(Pratkanis, 1995). We also include here psychothera- experimental clinical trials (clinical trial theory testing
pies, which are based on a variety of faiths or founda- [CTT]; see below the case of ‘‘specific efficacy’’). Sim-
tions that are outside the scientific approach, and often ilarly, in this framework, the therapeutic package is
do not seek scientific validation; we do not see pseudo- considered well supported if it has been empirically
scientific psychotherapies based on faith in pejorative validated at various levels in at least two randomized
terms, but rather different from what is considered clinical trials or equivalent designs (e.g., large series of
good scientific approach. single case experimental designs), by two different
Thus, as compared to the previous classification sys- investigators or investigating teams (I) (for details,
tems—APA ⁄ Division 12 ⁄ SSCP’s list of empirically val- including additional criteria of manualization and sam-
idated treatments—one of the most influential in the ple description and the issue of ‘‘equivalence of
field, (Chambless et al., 1998), we add the requirement designs,’’ see Chambless et al., 1998). The various
that there should also be a clear relationship between a levels of treatment package validation are discussed as
guiding psychological theoretical base and the empirical follows (see also Wampold, 2001):
data collected. We propose that in order for a thera-
peutic package to reach the highest level of evidence- (a) absolute efficacy (AE)—the therapeutic package
based support—EBP—both the therapeutic package is significantly better than a control condition.
and the underlying theory must be well supported by The control condition could be a no-treatment
scientific evidence. control condition (better than no treatment
[BNT]), but more often, a waiting-list control
Category I: Evidence-Based Psychotherapies condition (BWL), and ⁄ or
A Category I EBP has both a well-supported ⁄ well-vali- (b) relative efficacy (RE)—the therapeutic package
dated theory (e.g., supporting empirical data) and a is equivalent to or better than another evidence-
well-supported ⁄ well-validated therapeutic package based psychological intervention (equivalent to
(derived from the validated theory). Theory refers to standard treatment [EST]; better than standard
the mechanisms of change, namely the hypothesized treatment [BST]). That is, both the tested thera-
psychological factors involved in pathology and health, peutic package and the established psychotherapy

CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE • V18 N2, JUNE 2011 92

 should be better than control (BC) conditions Category II: Intervention-Driven Psychotherapies
(i.e., AE, coded as BNT or BWL) and ⁄ or at least An intervention-driven psychotherapy refers to a well-
as good as a standard treatment in the field (as supported therapeutic package but insufficiently sup-
good as standard treatment [GST]), and ⁄ or ported ⁄ investigated underlying theory from which that
(c) specific efficacy (SE)—involves meeting two package is derived. This does not mean that the theory
primary criteria. First, the therapeutic package is unscientific or wrong. It simply means that we do not
must be (a) significantly better than pill and ⁄ or yet have data regarding its validity (no data [ND]; if the
medical and ⁄ or psychological placebo (e.g., theory had been disproven, then the therapy would be
attention control; better than pill [BP]; and ⁄ or moved to Category V), or that we have preliminary data
better than psychological placebo [BPP]; and ⁄ or less than minimum standards (PD), or that we have
better than medical placebo), and ⁄ or (b) equiva- mixed data (MD); ND, PD, and MD should be used as
lent to active ⁄ standard psychological therapies codes during within-category analyses. The therapeutic
(EST) or significantly better than other package should be considered well supported according
active ⁄ standard psychotherapies (BST; see ‘‘rela- to the procedure described in Category I. Category II
tive efficacy’’ above). Second, the underlying roughly corresponds to the criteria established by Divi-
theory for specific mechanisms of change in case sion 12 of the APA ⁄ SSCP for their list of empirically
of the therapeutic package must be empirically validated treatments (Chambless et al., 1998; http://
supported by component analyses (CA) and ⁄ or www.psychology.sunysb.edu/eklonsky-/division12/).
patient by treatment interactions (PxT), and ⁄ or However, Category II has the advantage that lack of
analyses of mediation and ⁄ or moderation (MM) empirical support for underlying theory is made explicit.
(see also Wampold, 2001); it enhances the valid-
ity of the theory about the mechanisms of Category III: Theory-Driven Psychotherapies
change and ⁄ or is a direct proof of it. A Category III theory-driven psychotherapy refers to
the circumstance of a well supported theory (e.g., see
When an EBP passes—by statistical and clinical sig- Category I above), but an insufficiently supported ther-
nificance (see Pintea, 2010)—both the efficacy (e.g., apeutic package (no [ND], PD less than minimum
how it works in laboratory controlled conditions) and standards [PD], or MD); ND, PD, and MD should be
the effectiveness (e.g., how it works in real clinical set- used as codes during within category analyses. The
tings) tests, including empirical support for the underly- therapeutic package is still in need of empirical testing
ing theory as described above, it is an evidence-based in randomized clinical trials for clear documentation of
bona fide psychotherapy (EBBP). The codes men- absolute, relative, and ⁄ or specific efficacy.
tioned above (e.g., SE for ‘‘specific efficacy’’) give us
more information about EBPs or about psychotherapy Category IV: Investigational Psychotherapy
forms in the other categories (see Table 1 for an exam- A Category IV therapy is characterized by an insuffi-
ple of coding within-category analyses). ciently supported theory and an insufficiently supported
Within the context of the preceding material, it is therapeutic package. A Category IV therapy is still in
worth mentioning that the basic clinical skills— an early investigational phase, where ideas for both the-
common ⁄ contextual factors—(see Wampold, 2001), ory and therapeutic package are developing, but have
such as the therapeutic relationship, providing rationale not yet been tested (ND), or have been generated PD
(clinical conceptualization), treatment expectations, less than minimum standards (PD), or mixed results
etc., are important parts of any therapy, including the (MD); ND, PD, and MD should be used as codes dur-
EBP. They are parts of the theory about the mecha- ing within-category analyses. This is not to say that
nisms of change, together with the specific constructs either the theory or therapeutic package is somehow
of a certain psychotherapy (e.g., irrational beliefs in wrong or ineffective, but rather that the scientific test-
cognitive-behavioral psychotherapies [CBT]). ing and validation remain to be completed.

NEW EVALUATIVE FRAMEWORK FOR EVIDENCE-BASED PSYCHOTHERAPIES • DAVID & MONTGOMERY 93

Category V: Good Intervention and Bad Theory-Driven category analyses. This indicates that more theoretical
Psychotherapies clarifications are needed before running major trials to
A Category V psychotherapy is defined by a well- test the therapeutic package. If presented and used in
supported therapeutic package (see Category I) and a clinical practice as a form of scientifically based psycho-
theory about mechanisms of change for which SCE therapy, it would be an example of pseudoscientifically
exists. The theory should be considered as having SCE oriented psychotherapy.
within this framework if it has been empirically invali-
dated in at least two rigorous studies, by two different Category VIII: Bad Intervention-Driven Psychotherapies
investigators or investigating teams. It raises important A Category VIII psychotherapy is defined by equivocal
ethical questions and asks for guidelines about how to data (no [ND], PD less than minimum standards [PD],
frame this therapeutic package to clients to avoid the or MD) for its theory and SCEP; ND, PD, MD, and
meaning of the invalidated theory of change, if the cli- SCEP should be used as codes during within-category
nician decides to use it in lack of other available treat- analyses. If presented and used in clinical practice as a
ments. It can stimulate further research to find out the form of scientifically based psychotherapy, it is an
real mechanism of change for the already validated example of pseudoscientifically oriented psychotherapy.
treatment package. If presented and used in clinical
practice as a form of scientifically based psychotherapy, Category IX: Bad Theory and Bad Intervention-Driven
it would be an example of pseudoscientifically oriented Psychotherapies
psychotherapy. A Category IX psychotherapy is defined by SCE for its
theory (SCET) and SCEP. If presented and used in
Category VI: Good Theory and Bad Intervention-Driven clinical practice and scientific settings as a form of scien-
Psychotherapies tifically based psychotherapy, this would be a very good
A Category VI psychotherapy means that its theory is example of pseudoscientifically oriented psychotherapy.
well supported, but there is SCE for its therapeutic Table 1 synthesizes the above described categories.
package (SCEP). The therapeutic package should be
considered as having SCE within this framework if it IMPLICATIONS OF THE NEW EVALUATIVE FRAMEWORK FOR
has been empirically invalidated in at least two rigorous EVIDENCE-BASED PSYCHOTHERAPIES
studies, by two different investigators or investigating The proposed evaluative framework for evidence-based
teams. It might be the case that the therapeutic package psychotherapies has multiple implications.
was not correctly derived from the theory about the At the practical level, the proposed evaluative
mechanisms of change and ⁄ or its test was not properly framework addresses two current problems within the
designed (e.g., adherence to the protocol, therapists’ psychotherapy field. First, by adding the requirement
competency, etc.). If presented and used in clinical of testing theory to the evaluative framework, in addi-
practice as a form of scientifically based psychotherapy, tion to testing therapeutic packages, the framework
this should be a case of ethical concern, and it would rules out the possibility of designating interventions
be an example of pseudoscientifically oriented psycho- with false theoretical underpinnings as scientific psy-
therapy. However, it can stimulate further research to chotherapies. This change protects the integrity of the
develop new treatment packages based on the validated field of psychotherapy, and perhaps more importantly,
theory and test them. protects potential clients from bogus interventions. The
proposed classification framework defines POPs
Category VII: Bad Theory-Driven Psychotherapies (Categories V–IX), and it clearly distinguishes such
A psychotherapy included in Category VII is defined approaches from scientifically oriented approaches to
by no (ND), PD less than minimum standards (PD), or psychotherapy (SOPs—Categories I–IV). This frame-
MD regarding the efficacy and ⁄ or effectiveness of its work does not rule out the possibility of therapeutic
therapeutic package and SCE for its theory; ND, PD, packages moving from the POPs to the SOPs catego-
MD, and SCE should be used as codes during within- ries (or vice versa), but it highlights the necessary step

CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE • V18 N2, JUNE 2011 94

of adopting scientific methods for this to occur. In this VII) because although its theory was invalidated by a ser-
context, rather than talking about ‘‘scientific’’ versus ies of studies (for details, see Heap, 1988; Lilienfeld et al.,
‘‘pseudoscientific’’ psychotherapy, maybe it would be 2003), it continues to be promoted in practice based on
better to make and promote a distinction between the same theory, as if it were valid.
‘‘scientific (conventional) psychotherapy’’ (Categories An additional practical advantage of theoretically
I–IV) and alternative ⁄ complementary psychotherapy’’ informed interventions is that they allow clinical flexi-
(Categories V–IX), following a similar distinction in bility. Modern treatment manuals allow for tailoring of
medicine: ‘‘scientific (conventional) medicine’’ versus interventions to individual client needs and include a
‘‘alternative ⁄ complementary medicine’’ (for definitions, foundation based on common therapeutic factors (e.g.,
see http://nccam.nih.gov/health/whatiscam/#defining- therapeutic relationship, providing a rationale [clinical
cam). Obviously, what is alternative at a certain point can conceptualization], and setting appropriate treatment
become mainstream later (scientific ⁄ conventional), if we expectations). For example, within CBT treatment of
get data for its efficacy ⁄ effectiveness, following the model depression, theory guides clinicians to help clients
proposed in this article. This idea deserves to be explored change irrational beliefs into rational beliefs to alleviate
in future analyses. depressed mood. CBT theory (Beck, 1995) does
The history of hypnosis is an excellent example of prescribe specific techniques to do that, but it also pro-
the movement from POPs to SOPs. Many scholars attri- vides options (e.g., empirical, logical, pragmatic, and
bute the origins of hypnosis to mesmerism. In the 18th metaphorical disputations) on how to accomplish this
century, mesmerism was used to treat what we would goal based on individual patient needs and styles, all in
now refer to as hysterical symptoms in Austria and a therapeutic setting (e.g., therapeutic relationship, cog-
France. The theory underlying mesmerism was that nitive conceptualization ⁄ rationale, and positive treat-
‘‘animal magnetism’’ was responsible for beneficial ment expectations).
effects. Using an elegant experimental design, Ben At the theoretical level, the proposed framework
Franklin disproved the underlying theory. As flawed as has major implications for current debates within the
the underlying theory was, under an evaluative frame- field of psychotherapy. Two of the main aspects of
work that does not address the validity of theoretical the ‘‘evidence-based versus common factors’’ debate
underpinnings, mesmerism could today be considered a are discussed.
scientifically based psychotherapy. More modern studies First, championed by Wampold (2001), the common
of hypnosis highlight the development of a psychologi- factor movement within the field of psychotherapy
cal intervention according to the proposed evaluative takes the position that common factors (e.g., the thera-
system. From its roots in mesmerism (POPs), hypnosis peutic relationship, providing clinical rationale for dis-
has developed into a scientifically oriented psychological orders, and providing therapeutic strategies related to
intervention. There are now strong randomized clinical the clinical rationale) are responsible for most of the
trials to support its efficacy as a therapeutic package con- therapeutic change. Specific therapeutic factors (e.g.,
sistent with Category II (Lang et al., 2006; Montgomery changing irrational beliefs into rational ones in CBT)
et al., 2007; Schnur et al., 2008), as well as empirical are not viewed as responsible for an important thera-
data supporting the underlying theory (e.g., Montgom- peutic benefit. Thus, the underlying theory of change is
ery et al., 2010) consistent with Category III. Together, relegated to an irrelevant status, as there is only one the-
these data can place hypnosis squarely in Category I for ory of change—common factors, and all therapeutic
side effects of medical treatments, highlighting the packages can be viewed as equivalent in regard to effec-
implication of movement from Category IV to tiveness. Not surprisingly, the common factors move-
Category I as the scientific process is applied. ment is often at odds with the current EBP movement
An interesting shift from SOPs to POPs is illustrated (Chambless et al., 1998), which is striving to validate
by neurolinguistic programming. Once an interesting specific therapeutic packages and approaches. Overall,
system (e.g., Category IV of SOPs, according to our we agree that common factors have a clear role in the
classification), it is now seen largely as a POP (Category application and effectiveness of psychological interven-

NEW EVALUATIVE FRAMEWORK FOR EVIDENCE-BASED PSYCHOTHERAPIES • DAVID & MONTGOMERY 95

tions. Common factors are, in part, responsible for the underlying cognitive therapies (see David, 2004; David
benefits of psychological therapies (Wampold, 2001). & Szentagotai, 2006). If we understand this and we see
However, we also argue that specific therapeutic pack- the theory of the mechanisms of change as incorporat-
ages have demonstrated benefit above and beyond the ing both common and specific factors, then the debate
contribution of common factors (for discussions, see is futile and we can focus our effort on exploring the
Wampold, 2001). Some could say that this specific interactions between common and specific factors in
effect is small as compared to that of common factors. an empirically supported theory of mechanisms of
However, this is similar to the effect of an active sub- change (see also Ilardi & Craighead, 1994, for a similar
stance compared to a placebo in pharmacotherapy (e.g., argument in the case of CBT treatment for depres-
20% vs. 80% in case of fluoxetine; Kirsch, Moore, sion).
Scoboria, & Nicholls, 2002). The hope is that while Thus, the proposed evaluative framework can
placebo (or common factors in the psychotherapy field) accommodate the contribution of both specific and
has reached its maximum potential, the improvement common factors to both the therapeutic package and
for a large percentage of patients who do not or do the underlying theory. We believe that this framework
respond well to placebo (or common factors) will come lays an important foundation for the continued growth
from development of these specific factors (active sub- of applied and theoretical psychotherapy research and
stance in pharmacotherapy or specific psychological according to our model, the ‘‘evidence-based versus
mechanisms in psychotherapy). That is, the adherence common factors’’ debate is, at times, futile and a
to the scientific method has and will continue to potentially misguided one.
demonstrate the additive benefit of specific therapies. At the ethical level, the proposed framework also
Thus, as concerning this issue, the psychotherapy field has some implications. For example, a therapeutic
behaves in a manner consistent with that of the package validated in clinical trials could have (a) a
pharmacotherapy field. Simply put, the available data well-validated theory, (b) a theory with equivocal
support the position that both common and specific fac- evidence, or (c) an invalidated theory. While it would
tors contribute to the effectiveness of psychological be ethical of a clinician to use the hypothetical package
interventions, and both should be maximized to in clinical practice in conditions (1) and (2), it is
promote the greatest patient benefit. ethically problematic to use it in condition 3 (e.g.,
Second, according to the common factors move- ‘‘voodoo’’ and ⁄ or mesmerism like psychotherapy). An
ment, the research of the specific factors in psycho- analogy can also be made to old-time ‘‘snake oil’’ sales-
therapy (e.g., in cognitive therapies) is seen as less men. The snake oil might provide some benefit to its
relevant, by saying that noncognitive psychological users, but it is unethical to prescribe such a (placebo)
interventions and drug treatments, for example, change treatment when one clearly considers the underlying
cognitions in a manner indistinguishable from cogni- mechanism of change to be a sham.
tive therapy. Wampold (2001) stated that this detracts
from a specific ingredient argument for the efficacy of FURTHER CLARIFICATIONS OF THE NEW EVALUATIVE
cognitive therapy (or any other psychotherapy), as the FRAMEWORK FOR EVIDENCE-BASED PSYCHOTHERAPIES
therapeutic changes to cognitions could be achieved by When one proposes a new classification scheme, the
a variety of means. However, this argument appears to limitations and their implications should be discussed
be based on a false assumption regarding the unique explicitly. We focus on the most important ones.
and essential factors of cognitive therapy. That is, How do we define (a) ‘‘well-supported’’ theory
changes in cognitions are the mediator of cognitive and ⁄ or therapeutic package and (b) ‘‘SCE’’? Basically,
therapies that produce therapeutic benefit. This does all therapies have some findings that are less strong than
not mean that changes in cognitions are the exclusive others or mixed results. The fact that all therapies have
domain of cognitive therapies. Cognitions can be some findings that are less strong than others or mixed
altered incidentally, indirectly, or directly by many results is also true for the ‘‘classic’’ classification scheme
approaches, which is not contrary to cognitive theories (see Chambless et al., 1998). Therefore, we propose to

CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE • V18 N2, JUNE 2011 96

deal with this issue in a similar manner, by asking for ory, we refer to the mechanisms of change, namely the
explicit, minimum standards (see Chambless et al., hypothesized psychological factors involved in pathol-
1998). For example ‘‘well supported’’ within this frame- ogy and health, which are targeted by the therapeutic
work means that it has been empirically supported in at package. A specific treatment is more scientifically
least two rigorous studies, by two different investigators legitimate if it is derived from experimental psychopa-
or investigating teams. For theory, evidence could come thology research that has clarified the nature of the dis-
from (a) experimental studies and ⁄ or (b) clinical trials; order, and, thus, there should be a correspondence
thus, the theory can be tested independent of its thera- between the mechanisms of treatment (‘‘mecha-
peutic package (e.g., in experimental studies) and ⁄ or nism ⁄ theory of change’’) and the mechanisms of the
during CTT. We only extend these standards by disorder (‘‘theory of disorder’’). For example, if irratio-
including and applying them not only to the therapeutic nal beliefs are hypothesized to be involved in psycho-
package (as Chambless et al., 1998), but also to the the- pathology and rational beliefs in health states, then they
ory underlying the therapeutic package, with clear-cut should mediate the impact of CBT on various out-
positive consequences, as discussed above. In the same comes. Thus, the theory can be tested independent of
line of argument, ‘‘SCE’’ for theory and ⁄ or therapeutic its therapeutic package (e.g., ‘‘theory of the disorder’’
package means that it has been empirically invalidated in experimental studies: see, for example, the experi-
(in relationship to the advanced hypotheses and ⁄ or mental designs in the ‘‘emotional self-regulation’’ para-
objectives) in at least two rigorous studies, by two dif- digm) and ⁄ or during CTT (‘‘theory of change’’; see
ferent investigators or investigating teams. For example, David & Szentagotai, 2006). Only correlational studies
the lack of efficacy of a new treatment package, as com- (e.g., between irrational beliefs and psychopathology),
pared to a standard treatment, may be seen as (a) invali- be they clinical or analogue, are important, but adjunc-
dating evidence—if the hypotheses and ⁄ or the tive. The definitive test is related to CTT and ⁄ or
objectives postulated a better efficacy for the new treat- experimental studies, following the ‘‘theory of change’’
ment, or (b) validating evidence—if the hypotheses strategy. For example, in considering cognitive therapy
and ⁄ or the objectives argued for the same efficacy for for panic disorder, correlational studies reflecting the
the new and the standard treatment. relationship between catastrophic cognitions and panic
‘‘Equivocal data’’ is an interesting case in our classifi- would not be sufficient. Support for the cognitive the-
cation. We include three situations in this category: (a) ory of panic must demonstrate the relationship between
ND available (yet), (b) PD less than minimum standards a reduction in catastrophic cognitions and a reduction
for either ‘‘well-supported’’ or ‘‘SCE’’ conditions, or in panic (see Clark et al., 1999).
(c) MD. Other ‘‘classical’’ classification schemes (e.g., Sometimes there are competing and mutually exclu-
Chambless et al., 1998) would accept a therapeutic sive theories that data suggest support the same thera-
package with only one supportive clinical trial (i.e., PD peutic package. This would be a case that asks for a
less than minimum standards in our terms) as ‘‘probably ‘‘crucial experiment’’ in Popperian terms (Popper,
efficacious.’’ In our scheme, this case would fit one of 1959), but all could be considered valid theories until
the categories III, IV, or VII, depending on the status we run this crucial experiment.
of its theory. Thus, our system allows for more nuances, A form of psychotherapy could be analyzed in detail
and is more conservative, protecting the field and the by explicitly coding the evidence for its theory and ⁄ or
patients from including too easily in the league of therapeutic package (see Table 1). Although this may
evidence-based psychotherapies (or more general: evi- seem a very difficult and complex process, it is not
dence-based health-related interventions) potentially mandatory, and one can use only the general classifica-
dangerous therapeutic packages, designated as ‘‘probably tion, based on the nine categories (i.e., between-cate-
efficacious’’ based solely on one supporting clinical trial gory analyses). However, a within-category analysis can
(see the cases of ‘‘voodoo’’ and ‘‘mesmerism’’). be performed if this process is necessary and ⁄ or relevant
A criticism could also be related to the definition of in complex comparisons of various psychotherapies for
‘‘theory.’’ As we said before, when we talk about the- research, practice, and ⁄ or health insurance goals; in this

NEW EVALUATIVE FRAMEWORK FOR EVIDENCE-BASED PSYCHOTHERAPIES • DAVID & MONTGOMERY 97

case we suggest using a narrative description accompa- Update on empirically validated therapies, II. The Clinical
nying the coding profile (see Table 1). Psychologist, 51, 3–16.
Finally, we may have situations in which a theory Chambless, D. L., Sanderson, W. C., Shoham, V., Bennett
and ⁄ or a therapeutic package could simultaneously fit Johnson, S., Pope, K. S., Crits-Christoph, P., et al.
(1996). An update on empirically validated therapies. The
both the criteria for ‘‘well-validated’’ and ‘‘SCE’’ con-
Clinical Psychologist, 49, 5–18.
ditions. In this case, it is the responsibly of an expert
Clark, D. M., Salkovskis, P. M., Hackman, A., Wells, A.,
panel to analyze the data and to decide whether this is a
Ludgate, J., & Gelder, M. (1999). Brief cognitive therapy
case of MD or one of the two conditions has a stronger for panic disorder: A randomized clinical trial. Journal of
impact (e.g., depending on the quality of studies, etc.). Consulting and Clinical Psychology, 67, 583–589.
This ‘‘expert’’-based solution is used today by various David, D. (2004). Special issue on the cognitive revolution
international organizations dealing with evidence-based in clinical psychology: Beyond the behavioral approach-
psychotherapies (see the NICE procedures). conclusions: Toward an evidence-based psychology and
psychotherapy. Journal of Clinical Psychology, 4, 447–451.
CONCLUSIONS David, D., & Szentagotai, A. (2006). Cognition in cognitive
The consistent lack of consideration of underlying the- behavior psychotherapies. Clinical Psychology Review, 26,
ory is a significant weakness of current evaluative psy- 284–298.
Heap, M. (1988). Neuro-linguistic programming: An interim
chotherapy frameworks. To clarify the meaning of
verdict. In M. Heap (Ed.), Hypnosis: Current clinical, experi-
EBP, provide guidelines for evaluating psychological
mental and forensic practices (pp. 268–280). London: Croom
interventions, and promote growth in the field, we
Helm.
propose a new evaluative framework. The main contri- Ilardi, S. S., & Craighead, W. E. (1994). The role of
bution of this framework is to stress the importance of nonspecific factors in cognitive-behavior therapy for
underlying theory when defining evidence-based psy- depression. Clinical Psychology: Science and Practice, 6, 295–
chotherapies. Theory not only allows differentiation of 299.
SOPs and POPs, but also is the engine driving thera- Kirsch, I., Moore, T. J., Scoboria, A., & Nicholls, S. S.
peutic package development and improvement, and (2002). The emperor’s new drugs: An analysis of antide-
without it, we can easily drift down false paths. pressant medication data submitted to the US Food and
Drug Administration. Prevention & Treatment, 5, Article
ACKNOWLEDGMENTS
23. http://www.journals.apa.org/prevention/volume5/pre
0050023a.html.
We thank the editor and the two anonymous reviewers
Lang, E. V., Berbaum, K. S., Faintuch, S., Hatsiopoulou, O.,
whose suggestions significantly improved our manuscript, not
Halsey, N., Li, X., et al. (2006). Adjunctive self-hypnotic
only in its exposition, but also in its conceptualization (e.g.,
relaxation for outpatient medical procedures: A prospec-
by suggesting terms such as beneficence and malfeasance).
tive randomized trial with women undergoing large core
breast biopsy. Pain, 5, 155–164.
NOTE
Lilienfeld, S. O., Lynn, S. J., & Lohr, J. M. (2003). Science and
1. Although, recently, the APA added the label ‘‘contro-
pseudoscience in clinical psychology. New York: Guilford
versial’’ (a) to treatments that generate conflicting results or
Press.
(b) to efficacious treatments whose claims about why they
McNally, R. J. (1999). EMDR and mesmerism: A comparative
work are conflicting with the research evidence, this is not
historical analysis. Journal of Anxiety Disorders, 13, 225–236.
elaborated and ⁄ or considered an independent criterion, as we
Montgomery, G. H., Bovbjerg, D. H., Schnur, J. B., David,
propose in this article.
D., Goldfarb, A., Weltz, C. R., et al. (2007). A random-
ized clinical trial of a brief hypnosis intervention to control
REFERENCES side effects in breast surgery patients. Journal of the National
Beck, J. S. (1995). Cognitive therapy: Basic and beyond. New Cancer Institute, 99, 1304–1312.
York: Guilford Press. Montgomery, G. H., Hallquist, M. N., Schnur, J. B., David,
Chambless, D. L., Baker, M. J., Baucom, D. H., Beutler, D., Silverstein & J. H., Bovbjetg, D. H. (2010). Mediators
L. E., Calhoun, K. S., Crits-Christoph, P., et al. (1998). of a brief hypnosis intervention to control side effects in

CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE • V18 N2, JUNE 2011 98

 breast surgery patients: Response expectancies and emo- Schnur, J. B., Bovbjerg, D. H., David, D., Tatrow, K.,
tional distress. Journal of Consulting and Clinical Psychology, Goldfarb, A. B., Silverstein, J. H., et al. (2008). Hypnosis
78, 80–88. decreases presurgical distress in excisional breast biopsy
Pintea, S. (2010). The relevance of results in clinical research: patients. Anesthesia and Analgesia, 106, 440–444.
Statistical, practical, and clinical significance. Journal of Wampold, B. E. (2001). The great psychotherapy debate:
Cognitive and Behavioral Psychotherapies, 10, 101–115. Models, methods, and findings. Mahwah, NJ: Lawrence
Popper, K. (1959). Logic of scientific discovery. New York: Erlbaum.
Science Editions.
Pratkanis, A. (1995). How to sell a pseudoscience. Skeptical Received January 8, 2010; revised July 31, 2010; accepted
Inquirer, 19, 19–25. August 10, 2010.

NEW EVALUATIVE FRAMEWORK FOR EVIDENCE-BASED PSYCHOTHERAPIES • DAVID & MONTGOMERY 99