Bright Et Al. - 2012 - Effect of Clinical Decision-Support Systems A Sys PDF

Annals of Internal Medicine
REVIEW
Effect of Clinical Decision-Support Systems
A Systematic Review
Tiffani J. Bright, PhD; Anthony Wong, MTech; Ravi Dhurjati, PhD; Erin Bristow, BA;
Lori Bastian, MD, MS; Remy R. Coeytaux, MD, PhD;
Gregory Samsa, PhD; Vic Hasselblad, PhD; John W. Williams, MD, MHS; Michael D.
Musty, BA; Liz Wing, MA; Amy S. Kendrick, RN, MSN;
Gillian D. Sanders, PhD; and David Lobach, MD, PhD
Background: Despite increasing emphasis on the role of clinical

decision-support systems (CDSSs) for improving care and reducing
costs, evidence to support widespread use is lacking.
Purpose: To evaluate the effect of CDSSs on clinical outcomes,

health care processes, workload and efficiency, patient satisfaction,
cost, and provider use and implementation.
Data Sources: MEDLINE, CINAHL, PsycINFO, and Web of Science

through January 2011.
Study Selection: Investigators independently screened reports to

identify randomized trials published in English of electronic CDSSs
that were implemented in clinical settings; used by providers to aid
decision making at the point of care; and reported clinical, health
care process, workload, relationship-centered, economic, or pro-
vider use outcomes.
Data Extraction: Investigators extracted data about study design,

participant characteristics, interventions, outcomes, and quality.
Data Synthesis: 148 randomized, controlled trials were included.

A total of 128 (86%) assessed health care process measures, 29
(20%) assessed clinical outcomes, and 22 (15%) measured costs.
Both commercially and locally developed CDSSs improved health

care process measures related to performing preventive services
(n = 25; odds ratio [OR], 1.42 [95% CI, 1.27 to 1.58]), ordering
clinical studies (n = 20; OR, 1.72 [Cl, 1.47 to 2.00]), and prescrib-
ing therapies (n = 46; OR, 1.57 [Cl, 1.35 to 1.82]). Few studies
measured potential unintended consequences or adverse effects.
Limitations: Studies were heterogeneous in interventions, popula-

tions, settings, and outcomes. Publication bias and selective report-
ing cannot be excluded.
Conclusion: Both commercially and locally developed CDSSs are

effective at improving health care process measures across diverse
settings, but evidence for clinical, economic, workload, and effi-
ciency outcomes remains sparse. This review expands knowledge in
the field by demonstrating the benefits of CDSSs outside of expe-
rienced academic centers.
Primary Funding Source: Agency for Healthcare Research and
Quality.
Ann Intern Med. 2012;157:29-43.

For author affiliations, see end of text.
This article was published at www.annals.org on 24 April 2012.
www.annals.org
Despite increasing emphasis on clinical decision-

support systems (CDSSs) in improving care and re-
ducing costs, evidence supporting widespread use is lim-
ited. A CDSS is “any electronic system designed to aid
directly in clinical decision making, in which characteris-
tics of individual patients are used to generate patient-
specific assessments or recommendations that are then pre-
sented to clinicians for consideration” (1). This review
examines 3 types of decision-support interventions.
Classic CDSSs include alerts, reminders, order sets,

drug-dose calculations that automatically remind the clini-
cian of a specific action, or care summary dashboards that
provide performance feedback on quality indicators. Infor-
mation retrieval tools, such as an “infobutton” embedded
in a clinical information system, are designed to aid clini-
cians in the search and retrieval of context-specific knowl-
edge from information sources based on patient-specific
information from a clinical information system. Knowl-
edge resources, such as UpToDate, Epocrates, and MD
Consult, consist of distilled primary literature that allows
selection of content germane to a specific patient to facili-
tate decision making at the point of care or for a specific
care situation.
Until recently, most studies of CDSSs came from 4

benchmark settings (Brigham and Women’s Hospital/Part-
ners HealthCare, the Department of Veterans Affairs, LDS
Hospital/Intermountain Healthcare, and the Regenstrief
Downloaded From: http://annals.org/ on 10/31/2016
Institute) (2). Although several reviews have examined the

effects of CDSSs (1, 3-9), many questions remain about
their impact. This systematic review adds to the literature
by summarizing trials of CDSSs implemented in a clinical
setting to aid decision making at the point of care or for a
specific care situation.
METHODS
We developed and followed a standard protocol for

our review. Full details of our methods, search strategies,
results, and conclusions are provided in a report commis-
sioned by the Agency for Healthcare Research and Quality,
available at www.effectivehealthcare.ahrq.gov.
Data Sources and Searches
We searched for studies done between January 1976

and January 2011 in MEDLINE accessed through PubMed,
CINAHL, PsycINFO, and Web of Science.
Study Selection
We identified randomized trials of CDSSs imple-

mented in a real clinical setting and used by health care
providers to aid decision making at the point of care or for
a specific care situation. Studies had to report at least one
of the following types of outcomes: clinical (length of stay,
morbidity, mortality, health-related quality of life, and ad-
© 2012 American College of Physicians [ 29
REVIEW | Effect of Clinical Decision-Support Systems
verse events), health care process (recommended preventive

care, clinical study, or treatment ordered or completed), user
workload and efficiency (user knowledge, number of patients
seen, clinician workload, and efficiency), relationship-centered
(patient satisfaction), economic (cost and cost-effectiveness),
or use and implementation by a health care provider (ac-
ceptance, satisfaction, use, and implementation). We ex-
cluded studies that described nonelectronic CDSSs, in-
cluded fewer than 50 participants, were not published in
English, described closed-loop systems that did not involve
a provider, evaluated systems that required mandatory
compliance with the CDSS, or evaluated only the perfor-
mance of the system as opposed to its effect on clinical
practice.
Data Extraction and Quality Assessment
Data related to study setting and design, sample char-

acteristics, intervention characteristics, comparators, and
outcomes were extracted by 1 reviewer and confirmed by
another. Two reviewers used a standardized approach to
independently categorize the quality of individual studies
as good, fair, or poor (10) and evaluated the overall strength of
evidence for each outcome as high, moderate, low, or in-
sufficient (11). Reviewers also identified issues related to
study setting, interventions, and outcomes that limited ap-
plicability of evidence (10, 12).
Data Synthesis and Analysis
A priori—defined outcomes believed to be important

in measuring the effect of CDSSs in improving clinical
practice guided our synthesis process. Studies with a com-
mon outcome were grouped together to facilitate qualita-
tive analysis. Quantitative analysis was done where 4 or
more studies assessed the same outcome in the same man-
ner, regardless of the specific CDSS intervention. Sum-
mary estimates were calculated by using the DerSimonian
and Laird (13) random-effects model as implemented in
Comprehensive Meta-analysis, version 2.2.055 (Biostat,
Englewood, New Jersey).
Role of the Funding Source
Primary funding was provided by the Agency for

Healthcare Research and Quality. The funding source for-
mulated the initial study questions but otherwise had no
role in the design, analysis, or interpretation of the data or
in the decision to submit the manuscript for publication.
RESULTS
We screened 15 176 abstracts, evaluated 1407 full-text

articles, and included 160 articles, representing 148 unique
studies (Appendix Figure, available at www.annals.org).
Appendix Table 1 (available at www.annals.org) summa-
rizes important characteristics of the studies and their qual-
ity rating. A total of 128 studies (86%) assessed health care
process measures, 29 (20%) assessed clinical outcomes, and
30 | 3 July 2012 |Annals of Internal Medicine | Volume 157 ® Number 1
22 (15%) measured costs. Many studies (7 = 51) were

performed in environments with established health infor-
mation technology (IT); many were multisite studies in-
volving multiple institutions (7 = 40).
The Table summarizes the most important findings

and the strength of supporting evidence for those findings,
whereas Appendix Table 2 (available at www.annals.org)
provides examples of interventions that assessed the out-
comes of interest. Both commercially and locally developed
CDSSs improved health care process measures related to
performing preventive services, ordering clinical studies,
and prescribing therapies. Few studies (z = 15) indicated
conceptualization of potential unintended consequences or
measured potential adverse effects of implementing decision-
support tools.
Clinical Outcomes
Morbidity
Several studies assessed morbidity outcomes (14-38),

such as hospitalizations, Apgar scores, surgical site infec-
tions, cardiovascular events, colorectal cancer, deep venous
thrombosis, and hypoglycemia events. Topics addressed in-
cluded diagnosis (16, 21, 29, 30, 34), pharmacotherapy
(15, 18-22, 25, 28, 33), chronic disease management (14,
19, 20, 22, 26-28, 31, 32, 36, 38), laboratory test order-
ing (19, 37), immunizations (19, 35), preventive care (17,
19, 28-30, 37), and discharge planning (23, 24). Approx-
imately 50% of the studies were performed in an academic
setting.
Many studies evaluated locally developed interventions

implemented in the ambulatory environment. Typical in-
terventions were automatically delivered, system-initiated
recommendations provided synchronously at the point of
care to enable decision making during the health care
provider—patient encounter. Three such interventions re-
quired a mandatory response (that is, required that the user
respond to the given recommendation, whether that re-
sponse was to accept or dismiss the recommendation or to
modify the user’s action) (17, 18, 37).
Comparators included usual care or no CDSS and the

same CDSS with additional features. Limitations included
short follow-up, low statistical power to detect important
differences, and the potential for contamination of provid-
ers in control groups that improved because of knowledge
of interventions.
Meta-analysis of these heterogeneous studies (7 = 10)

suggested that CDSSs improved morbidity outcomes (rel-
ative risk, 0.88 [95% CI, 0.80 to 0.96]). We rated this
level of evidence as moderate. Most studies were good
quality, and many of the interventions were evaluated in
multiple institutions. However, the interventions were of-
ten paper-based or standalone systems implemented in ac-
ademic or Veterans Affairs settings and were targeted to-
ward a single condition.
www.annals.org
Effect of Clinical Decision-Support Systems REVIEW
Table. Summary of Evidence, by Outcome
Outcome Evidence Studies (Quality Meta-analysis Studies

Strength Rating), n Result for Included
Outcomes in the Meta-
(95% ClI) analysis, n
Clinical outcomes
Length of stay Low 6 (6 good) RR, 0.96 5
(0.88-1.05)
favoring CDSS
Morbidity Moderate 22 (13 good, 7 fair, RR, 0.88 16
2 poor) (0.80-0.96)
favoring CDSS
Mortality Low 7 (6 good, 1 fair) OR, 0.79 6

(0.54-1.15)
favoring CDSS
HRQOL Low 6 (3 good, 2 fair, NA -

1 poor)
Adverse events Low 5 (3 good, 1 fair, RR, 1.01 5

1 poor) (0.90-1.14)
favoring control
Health care process measures
Recommended preventive High 43 (20 good, OR, 1.42 25

care service ordered or 16 fair, 7 poor) (1.27-1.58)
completed favoring CDSS
Recommended clinical study Moderate 29 (16 good, 9 fair, OR, 1.72 20

ordered or completed 4 poor) (1.47-2.00)
favoring CDSS
Recommended treatment High 67 (35 good, OR, 1.57 46

ordered or prescribed 24 fair, 8 poor) (1.35-1.82)
favoring CDSS
User workload and efficiency outcomes

Effect on user knowledge Insufficient 5 (4 fair, 1 poor) NA =
Number of patients seen Insufficient 0 NA -

per unit time
Clinician workload Insufficient 0 NA =
Efficiency Low 7 (3 good, 4 fair) NA -
Relationship-centered outcomes
Other Substantial Findings
Limited evidence that CDSSs that automatically delivered

system-initiated recommendations to providers were
effective or demonstrated a trend toward reducing
length of stay
Modest evidence from academic and community

inpatient and ambulatory settings that locally
developed CDSSs that automatically delivered
system-initiated recommendations to providers
synchronously at the point of care were effective or
demonstrated a trend toward reducing patient
morbidity
Limited evidence that CDSSs integrated in CPOE or EHR

systems that automatically delivered system-initiated
recommendations to providers were effective or
demonstrated a trend toward reducing patient
mortality
Limited evidence from ambulatory settings that

locally developed CDSSs that automatically delivered
system-initiated recommendations to providers
demonstrated a trend toward higher HRQOL
scores
Limited evidence from academic settings that CDSSs that

delivered recommendations to providers synchronously
at the point of care demonstrated an effect on
reducing or preventing adverse events
Strong evidence from studies conducted in academic,

VA, and community inpatient and ambulatory settings
that locally and commercially developed CDSSs that
automatically delivered system-initiated recommend-
ations to providers synchronously at the point of care
and did not require a mandatory clinician response
were effective at improving the appropriate ordering
of preventive care procedures
Modest evidence from studies conducted in academic

and community inpatient and ambulatory settings that
CDSSs integrated in CPOE or EHR systems and locally
and commercially developed CDSSs that automatically
delivered system-initiated recommendations to
providers synchronously at the point of care and did
not require a mandatory clinician response were
effective at improving the appropriate ordering of
clinical studies
Strong evidence from academic, community, and VA

inpatient and ambulatory settings that locally and
commercially developed CDSSs integrated in CPOE or
EHR systems that automatically delivered system-
initiated recommendations to providers synchronously
at the point of care and did not require a mandatory
clinician response were effective at improving
appropriate treatment ordering or prescribing
None
None
None
Limited evidence that CDSSs demonstrated a trend
toward improving efficiency
Patient satisfaction Insufficient 6 (4 good, 1 fair, NA - Limited evidence that

clinician use of CDSSs had a
1 poor) positive effect on patient satisfaction
Continued on following page
www.annals.org 3 July 2012 | Annals of Internal Medicine [ Volume 157 ¢ Number 1|31
Table—Continued
Outcome Evidence Studies (Quality Meta-analysis
Strength Rating), n Result for
Outcomes
(95% Cl)
Economic outcomes
Cost Moderate 22 (10 good, 7 fair, NA
5 poor)
Cost-effectiveness Insufficient 6 (1 good, 5 fair) NA
Use and implementation outcomes
Health care provider Low 24 (9 good, 11 fair, NA

acceptance 4 poor)
Health care provider Moderate 19 (9 good, 7 fair, NA

satisfaction 3 poor)
Health care provider use Low 17 (5 good, 10 fair, NA
2 poor)
Implementation Insufficient 5 (3 fair, 2 poor) NA
Studies
Included
in the Meta-
analysis, n
Other Substantial Findings
- Modest evidence from academic and community

inpatient and ambulatory settings that locally and
commercially developed CDSSs integrated in CPOE or
EHR systems that automatically delivered system-
initiated recommendations to providers synchronously
at the point of care demonstrated a trend toward
lower treatment costs, total costs, and greater cost
savings
- Conflicting evidence from ambulatory setting about the

cost-effectiveness of CDSSs that delivered recommend-
ations to providers synchronously at the point of care
- Many successful CDSS studies did not report acceptance
= CDSSs that fostered high satisfaction among providers

were implemented within academic, community, and
VA ambulatory settings; were integrated in CPOE or
EHR systems; were locally and commercially
developed; automatically delivered system-initiated
recommendations to providers synchronously at the
point of care; and did not require a mandatory
clinician response
- Most included studies documented low use (<50% of
the clinicians’ time or of patient visits), or less than
50% of clinicians used the CDSS or received alerts to
guide therapeutic action
- Insufficient evidence for how CDSSs affected

implementation in practice, and no good-quality
studies specifically examined this outcome
CDSS = dlinical decision-support system; CPOE = computerized physician order entry;

EHR = electronic health record; HRQOL = health-related quality of life; NA =
not applicable; OR = odds ratio; RR = relative risk; VA = Veterans Affairs.
Mortality
Seven studies (17, 20-22, 33, 39, 40) reported mor-

tality outcomes. Issues addressed included diagnosis (21),
pharmacotherapy (20-22, 33, 40), chronic disease man-
agement (20, 22), preventing deep venous thrombosis
(17), and detecting and notifying clinicians of critical lab-
oratory values (39). Most CDSSs were locally developed
and integrated into a computerized physician order entry
(CPOE) or electronic health record (EHR) system and
had system-initiated recommendations delivered syn-
chronously at the point of care that did not require a
clinician response.
Interventions were evaluated against usual care or no

CDSS, except for 2 studies (20, 22) that compared the
same intervention with additional features. Limitations in-
cluded small sample size, duration shorter than 1 year, and
possible contamination of control providers.
Meta-analysis of these heterogeneous studies (7 = 0)

suggested no significant effect of CDSSs on mortality, al-
though Cls were wide (odds ratio [OR], 0.79 [CI, 0.54 to
1.15]). Of note, 2 studies reported a significant reduction
in mortality with use of CDSSs (20, 22). We rated the
overall strength of evidence related to mortality outcomes
as low. Most studies were conducted in a single academic

or Veterans Affairs setting with a comprehensive, well-
established health IT infrastructure.
Adverse Events
Five studies assessed the effectiveness of CDSSs in re-

ducing or preventing adverse events (23, 24, 39-42).
Studies were mostly implemented in an academic, inpa-
tient setting. Studies evaluated the effect of these interven-
tions to improve the timing of warfarin therapy (41), im-
prove discharge planning (23, 24), prevent adverse drug
events (42), detect critical laboratory values (39), and de-
tect potentially inappropriate or inadequate antimicrobial
therapy (40).
Typical interventions were locally developed, were in-

tegrated into a CPOE or an EHR system, and automati-
cally delivered system-imitated recommendations in real
time to enable decision making during the provider—
patient encounter. Only 1 study clearly required a manda-
tory response (39).
All of the CDSSs were evaluated against usual care or

no CDSS. Limitations included evaluation at a single in-
www.annals.org
stitution, evaluation periods less than 1 year, and potential

improvement in physician performance because of their
knowledge of the intervention.
Meta-analysis of these heterogeneous studies estimated

a relative risk of 1.01 (CI, 0.90 to 1.14), and neither this
summary nor any individual studies demonstrated a signif-
icant effect. We rated this level of evidence as low. Most
studies were good quality, and 2 were conducted at multi-
ple sites; however, these interventions primarily contained
locally developed knowledge, and results may not be gen-
eralizable to nonteaching settings.
Health Care Process Measures

Recommended Preventive Care Service Ordered or Completed
Forty-three studies examined the effect of CDSSs on

the rates of ordering or completing recommended preven-
tive care services (17, 19, 27, 28, 35, 37, 43—-82). Most
studies were conducted in the academic or ambulatory en-
vironment. Topics addressed included diagnosis (43, 47,
62, 71, 82), pharmacotherapy (19, 28, 49, 53, 54, 70, 77,
80), chronic disease management (19, 26, 28, 43, 44, 48,
51, 70, 72, 74, 76), laboratory test ordering (19, 37, 55,
60, 70, 71, 74, 75, 80, 81), preventive care (17, 19, 28, 37,
43, 45, 48, 52-60, 62-65, 68-71, 73-75, 79-82), im-
munizations (19, 35, 48-50, 52, 56, 66, 67, 78, 79, 81),
and initiating discussions with patients (60, 61, 72).
Most interventions were locally developed, paper-

based, or standalone systems and automatically delivered
recommendations in real time to enable decision making
during the health care provider—patient encounter. Only 7
of the interventions required a mandatory response (17,
37, 47, 49) or justification (64, 65, 68, 69, 75) for not
adhering to the recommendation.
Comparators included usual care or no CDSS, direct

comparison against the same CDSS with additional fea-
tures, or comparison of the same CDSS for different con-
ditions. Limitations included sparse data measuring patient
or economic outcomes; few assessments of long-term out-
comes of interventions; and the Hawthorne effect, which
probably stimulated more comprehensive preventive care
across groups.
In meta-analysis of these 25 heterogeneous studies (17,

27, 28, 35, 37, 43—63), the effect of CDSSs on preventive
care services was significant (OR, 1.42 [CI, 1.27 to 1.58])
(Figure 1). We rated this level of evidence as high. Approx-
imately one half of the studies were good quality, one third
were evaluated in multicenter trials, and one fourth addressed
multiple clinical conditions. However, most CDSSs were lo-
cally developed, not integrated into a CPOE or an EHR, and
evaluated in academic medical centers, all of which can affect
the generalizability of these findings.
Recommended Clinical Study Ordered or Completed

Twenty-nine studies evaluated the effect of CDSSs on
ordering and completing recommended clinical studies
(26, 31, 32, 83-109). Many studies were conducted in the
www.annals.org
academic setting, and most were evaluated in the ambula-

tory environment. Topics addressed included diagnosis
(85, 89-91, 95, 98, 101-103), pharmacotherapy (94, 98),
chronic disease management (26, 31, 32, 84, 85, 103),
laboratory test ordering (83, 87, 89, 92-94, 96, 97, 99—
101, 104-108), initiating discussions with patients (108,
109), and additional clinical tasks (86, 88, 90, 109).

tegrated CDSS recommendations in a CPOE or an EHR
system, and automatically delivered system-initiated recom-
mendations to enable decision making during the provider—
patient encounter. Eight interventions required a manda-
tory response (90, 99, 100, 104, 106, 108) or justification
(83, 105) for not adhering to the recommendation.

comparison against the same CDSS with additional fea-
tures, or comparison of the same CDSS for different con-
ditions. Limitations of the evidence base included diverse
metrics to assess adherence to ordering and completing a
recommended action, studies not designed to evaluate the
clinical or economic outcomes associated with the CDSS
interventions, and limited evidence of the effect of CDSSs
on a broad set of conditions.
Meta-analysis of these 20 heterogeneous studies (26,

31, 32, 83, 84, 88, 89, 91, 92, 94-96, 98—-103, 105, 108,
109) found a significant effect of CDSSs on ordering or
completing of clinical studies (OR, 1.72 [CI, 1.47 to
2.00]) (Figure 2). We rated this level of evidence as mod-
erate. Most studies were good quality, approximately one
third were implemented in multiple sites, and almost one
fourth included a direct comparison of the effectiveness of
the CDSS against the same intervention with additional
features. However, we noted a strong suggestion of publi-
cation bias in these studies, and these results may not be
generalizable to all settings because most studies were ei-
ther evaluated in environments with a well-established
health IT infrastructure or conducted outside of the
United States.
Recommended Treatment Ordered or Prescribed
Sixty-seven studies evaluated the effect of CDSSs on

ordering and prescribing therapy (14, 18, 20-22, 25-28,
33, 36, 38—-41, 43, 44, 53, 54, 70, 80, 82, 84, 88, 94, 98,
110-154). Many studies were conducted in the academic
setting, and most were evaluated in the ambulatory envi-
ronment. Topics addressed included diagnosis (21, 43, 82,
98, 112, 123, 129, 138, 151, 152), pharmacotherapy (18,
20-22, 25, 28, 33, 40, 53, 54, 70, 80, 94, 98, 110, 111,
114, 116, 117, 119, 122, 123, 125-134, 136, 138-150,
153, 154), laboratory test ordering (70, 80, 94, 110, 130),
chronic disease management (14, 20, 22, 26-28, 36, 38,
43, 44,70, 84, 110, 112, 113, 115, 118, 120, 121, 123—
125, 132, 133, 135, 141), preventive care (28, 43, 53, 54,
70, 80, 82, 120, 148), and additional clinical tasks (39, 41,
82, 88, 110, 137, 151, 152).
3 July 2012 |Annals of Internal Medicine | Volume 157 ® Number 1|33
Figure 1. Results of studies that examined whether recommended preventive care

services were ordered.
Author, Year (Reference)
McDowell et al, 1986 (56)

Chambers et al, 1989 (63)
McDonald et al, 1992a (35)
McDonald et al, 1992b (35)
McDonald et al, 1992¢ (35)
Litzelman et al, 1993 (55)
Overhage et al, 1996 (58)
Burack et al, 1998 (45)
Taylor et al, 1999 (60)
Cannon and Allen, 2000 (47)
Demakis et al, 2000 (48)
Dexter et al, 2001b (49)
Dexter et al, 2001a (49)
Eccles et al, 2002 (51)
Burack et al, 2003a (46)
Burack et al, 2003b (46)
Frank et al, 2004a (52)
Frank et al, 2004b (52)
Frank et al, 2004c (52)
Frank et al, 2004d (52)
Frank et al, 2004e (52)
Frank et al, 2004f (52)
Frank et al, 2004g (52)
Frank et al, 2004h (52)
Frank et al, 2004i (52)
Dexter et al, 2004 (50)
Apkon et al, 2005 (43)
Kucher et al, 2005 (17)
Price, 2005 (59)
Tierney et al, 2005 (26)

Fretheim et al, 2006 (53, 54)
Unrod et al, 2007 (61)
Bertoni et al, 2009 (44)
Gilutz et al, 2009 (28)
Sequist et al, 2009 (37)
Dykes et al, 2010 (62)
Odds Ratio (95% Cl)
8.856 (5.809-13.501)
1.356 (1.053-1.745)
1.204 (0.739-1.963)
2.598 (2.164-3.119)
1.899 (1.598-2.257)
2.260 (1.861-2.743)
1.390 (1.247-1.549)
0.949 (0.754-1.193)
1.208 (0.994-1.469)
3.435 (1.918-6.151)
4.090 (1.320-12.671)
1.569 (1.466-1.679)
1.502 (1.380-1.634)
2.038 (1.859-2.234)
0.964 (0.622-1.492)
1.445 (1.207-1.730)
0.967 (0.823-1.136)
1.894 (1.712-2.095)
1.715 (1.138-2.585)
1.284 (0.830-1.987)
1.120 (0.907-1.383)
1.093 (0.936-1.276)
1.029 (0.916-1.156)
0.979 (0.647-1.480)
0.951 (0.774-1.168)
0.890 (0.728-1.090)
0.752 (0.563-1.006)
1.222 (1.071-1.394)
2.965 (2.437-3.607)
2.975 (1.191-7.431)
0.916 (0.503-1.667)
1.218 (0.932-1.592)
0.640 (0.131-3.120)
0.931 (0.833-1.041)
1.277 (1.166-1.399)
1.073 (1.017-1.133)
1.318 (0.956-1.817)
1.415 (1.267-1.579)
Odds Ratio (95% Cl)
T T
0.01 0.1 10.0 100.0
Favors Control Favors CDSS
Studies reporting the odds ratio of adhering to recommendations for ordering or

completing preventive care services of CDSS vs. control groups. In the
25 studies comparing CDSS with control groups, the random-effects—combined odds
ratio of adherence to preventive care recommendations was 1.42
(95% CI, 1.27 to 1.58). CDSS = clinical decision-support system.

www.annals.org

tegrated into a CPOE or an EHR system, and automati-
cally delivered system-initiated recommendations in real
time to enable decision making during the provider—
patient encounter. Eighteen CDSSs required a mandatory
response (18, 39, 119, 122, 134, 139-143, 147, 148, 151,
152) or justification (112, 113, 136, 137, 145) for not
adhering to the recommendation. Limitations included in-
adequate follow-up periods to observe sustained results,
sparse data demonstrating how changes in clinician order-
ing and prescribing led to improvements in clinical or eco-
nomic outcomes, and study designs that did not capture
the extent to which nonadherence with recommended

therapy resulted in adverse events.
Meta-analysis of 46 heterogeneous studies (14, 18,

20-22, 25-28, 33, 36, 38, 40, 43, 44, 53, 54, 70, 82, 84,
94, 98, 110, 112-117, 121-124, 129, 130, 134-136,
141-144, 146, 148, 151-153) showed that intervention
providers with decision support were more likely to order
the appropriate treatment or therapy (OR, 1.57 [CI, 1.35
to 1.82]) (Figure 3). We rated this level of evidence as
high. Most studies were good quality, and most were eval-
uated in muldisite trials. However, generalizability may be
limited because most studies were implemented in the am-
Figure 2. Results of studies that examined whether recommended clinical studies

were ordered.
McDonald, 1976 (94)

Bates et al, 1999 (83)
Flottorp et al, 2002a (88)
Flottorp et al, 2002b (88)
Greiver et al, 2005a (89)
Greiver et al, 2005b (89)
Raebel et al, 2005 (99)
Palen et al, 2006 (96)
Wilson et al, 2006 (109)
Roukemia et al, 2008 (101)
Lee et al, 2009 (91)
Lo et al, 2009 (92)
Roy et al, 2009 (102)

Schriefer et al, 2009 (103)
Sundaram et al, 2009 (105)
Bell et al, 2010a (84)
Bell et al, 2010b (84)

Khan et al, 2010a (31)
Khan et al, 2010b (31)
Khan et al, 2010c (31)
Khan et al, 2010d (31)
Player et al, 2010 (98)
Walker et al, 2010 (108)
Odds Ratio (95% Cl)
4.640 (3.197-6.734)
1.930 (1.395-2.670)
2.870 (2.180-3.779)
1.100 (1.004-1.205)
0.810 (0.729-0.899)
2.370 (0.833-6.744)
2.040 (0.492-8.461)
1.600 (1.439-1.779)
1.020 (0.278-3.738)
0.980 (0.941-1.021)
1.280 (1.179-1.389)
1.460 (0.628-3.392)
5.860 (2.828-12.142)
12.540 (6.481-24.264)
1.070 (0.935-1.224)
3.450 (2.800-4.250)
2.070 (1.314-3.260)
1.880 (1.373-2.575)
1.160 (0.894-1.506)
15.290 (3.752-62.301)
1.170 (0.798-1.716)
1.390 (1.077-1.794)
1.400 (1.063-1.845)
1.740 (1.128-2.685)
1.330 (1.132-1.563)
1.270 (1.111-1.452)
1.716 (1.472-2.001)
Odds Ratio (95% CI)
0.01 0.1 1.0 10.0 100.0


completing recommended clinical studies of CDSS vs. control groups.
In the 20 studies comparing CDSS with control groups, the random-effects—combined
odds ratio of adherence to clinical study recommendations was
1.72 (95% CI, 1.47 to 2.00). CDSS = clinical decision-support system.
www.annals.org 3 July 2012 | Annals of Internal Medicine | Volume 157 ¢ Number 1|35
Figure 3. Results of studies that examined whether recommended treatments were

ordered.
McDonald, 1976 (94)

Vissers et al, 1995 (152)
Overhage et al, 1997 (130)
Rossi and Every, 1997 (136)
McCowan et al, 2001 (14)
Ansari et al, 2003 (22)
Filippi et al, 2003 (117)
Tamblyn et al, 2003 (144)
Zanetti et al, 2003 (18)
Weir et al, 2003 (153)
Krall et al, 2004 (122)

Murray et al, 2004 (36)
Subramanian et al, 2004 (38)
Apkon et al, 2005 (43)
Cobos et al, 2005 (113)
Rood et al, 2005 (135)
Feldstein et al, 2006 (87)
Fretheim et al, 2006 (53, 54)
McGregor et al, 2006 (40)
Montgomery et al, 2000 (129)
Paul et al, 2006 (21)
Roumie et al, 2006 (20)
Davis et al, 2007 (114)
Heidenreich et al, 2007 (25)
Hicks et al, 2008 (121)
Smith et al, 2008 (141)
van Wyk et al, 2008 (82)

Bertoni et al, 2009 (44)
Field et al, 2009 (116)
Gill et al, 2009 (70)
Gilutz et al, 2009 (28)

Linder et al, 2009 (123)
Locatelli et al, 2009 (124)
Tamblyn et al, 2010 (146)
Terrell et al, 2009 (147)
Bell et al, 2010a (84)
Bell et al, 2010b (84)

Bourgeois et al, 2010 (110)
Brier et al, 2010 (33)
Co et al, 2010 (112)
Player et al, 2010 (98)

Strom et al, 2010 (142)
Strom et al, 2010 (143)
Terrell et al, 2010 (148)
Odds Ratio (95% CI)

0.426 (0.221-0.821)
4.247 (1.398-12.901)
3.074 (1.280-7.381)
45.570 (6.636-312.940)
1.684 (1.078-2.631)
0.490 (0.197-1.219)
1.356 (1.207-1.523)
1.202 (1.089-1.327)
1.059 (0.604-1.856)
3.113 (1.896-5.111)
0.984 (0.512-1.892)
3.417 (2.637-4.428)
0.867 (0.518-1.452)
1.137 (0.833-1.552)
0.790 (0.554-1.126)
2.100 (1.641-2.687)
1.904 (1.679-2.159)
1.082 (0.829-1.412)
16.780 (6.742-41.764)
1.680 (1.405-2.009)
2.389 (1.959-2.913)
1.324 (0.885-1.980)
1.470 (1.030-2.098)
0.844 (0.626-1.137)
1.086 (0.464-2.541)
1.457 (1.145-1.855)
0.830 (0.739-0.932)
1.441 (0.975-2.130)
1.277 (0.696-2.343)
7.309 (5.979-8.935)
1.041 (0.655-1.654)
1.548 (1.095-2.188)
1.386 (1.002-1.918)
1.246 (1.137-1.366)
1.864 (1.208-2.875)
0.723 (0.511-1.022)
1.461 (1.162-1.836)
1.818 (1.124-2.942)
0.876 (0.723-1.062)
2.675 (2.098-3.410)
1.430 (1.161-1.761)
0.977 (0.552-1.729)
2.083 (1.384-3.133)
1.110 (0.861-1.431)
8.559 (4.936-14.842)
1.160 (0.877-1.535)
3.839 (1.716-8.589)
1.570 (1.352-1.823)
Odds Ratio (95% ClI)
I I I I
0.01 0.1 10.0 100.0

prescribing treatment of CDSS vs. control groups. In the 46 studies
comparing CDSS with control groups, the random-effects—combined odds ratio of
adherence to treatment recommendations was 1.57 (95% CI, 1.35
to 1.82). CDSS = clinical decision-support system.

www.annals.org
bulatory environment, were evaluated in settings where cli-

nicians were experienced EHR users or provided care in an
established health IT infrastructure, and incorporated
knowledge that was targeted toward specific conditions.
User Workload and Efficiency Outcomes
Evidence on the effect of CDSSs on clinician knowl-

edge or improved confidence in managing patient care was
insufficient (71, 72, 86, 155, 156). Seven studies examined
the effect of CDSSs on efficiency (23, 24, 40, 106, 141,
155-157). Limitations included contamination of clini-
cians in the control group that improved because of knowl-
edge of the intervention, evaluation periods that were too
brief to demonstrate an effect on efficiency, and small cli-
nician sample sizes.
We rated the level of evidence as low. Most interven-

tions contained locally developed knowledge, such as pro-
tocols or algorithms derived on the basis of local perfor-
mance, quality, and outcome data not representative of
other sites, and were evaluated in academic settings.
Economic Outcomes
Cost
Twenty-two studies reported costs (21, 26, 27, 31, 32,

36, 40, 43, 53, 54, 71, 83, 90, 1006, 109, 113, 118, 130,
141, 158-163). Objectives of the CDSSs included diagno-
sis (21, 43, 71, 90, 160), pharmacotherapy (21, 40, 53, 54,
130, 141), chronic disease management (26, 27, 31, 32,
36, 43, 113, 118, 141, 161, 162), laboratory test ordering
(71, 83, 106, 130, 160, 163), preventive care (43, 53, 54,
71, 158, 159), initiating discussions with patients (109,
159), and additional clinical tasks (90, 109). One study
reported reduced hospitalization expenses with CDSS use
(31, 32), and 12 studies reported that use had a positive
effect on costs compared with control groups and other
non-CDSS groups (21, 27, 40, 53, 54, 83, 90, 106, 113,
130, 141, 159, 160).
Modest evidence from academic and community inpa-

tient and ambulatory settings showed that locally and com-
mercially developed CDSSs had lower treatment costs, to-
tal costs, and reduced costs compared with control groups
and other non-CDSS intervention groups. Most studies
were conducted in the academic ambulatory setting and
evaluated locally developed, integrated CDSSs in CPOE or
EHR systems that automatically delivered system-initiated
recommendations synchronously at the point of care and
did not require a mandatory clinician response.
Cost-Effectiveness
Six studies (53, 54, 56, 57, 78, 95, 96, 161, 162)
examined the cost-effectiveness of CDSSs or their effect on
cost-effectiveness of care. These demonstrated conflicting
findings, with 3 studies suggesting that CDSSs were cost-
effective (53, 54, 78, 95) and 3 reporting that CDSSs were
not cost-effective (56, 57, 161, 162). Objectives included
diagnosis (95), pharmacotherapy (53, 54), chronic disease
www.annals.org
management (161, 162), preventive care (53, 54, 56, 57),

and immunizations (56, 78).
Use and Implementation Outcomes
Twenty-four studies assessed the effect of provider ac-

ceptance of CDSSs (19, 41, 55, 62, 75, 90, 105, 113, 119,
120, 136, 137, 145-147, 149, 158, 159, 164-170). Top-
ics addressed included diagnosis (62, 90, 166), pharmaco-
therapy (19, 119, 136, 145-147, 149, 164, 165), chronic
disease management (19, 113, 120, 168-170), laboratory
test ordering (19, 55, 75, 105), preventive care (19, 55, 62,
75, 120, 147, 158, 159, 167), immunizations (19), initi-
ating discussions with patients (159), and additional clini-
cal tasks (41, 90, 137).
Comparators included usual care or no CDSS and di-

rect comparison with the same CDSS with additional fea-
tures. One half of the studies required a mandatory re-
sponse (55, 90, 119, 147, 166) or justification (75, 105,
113, 120, 136, 137, 145) for not adhering to the recom-
mendation; however, there was no significant effect on
provider acceptance. Limitations included an inconsistent
definition of provider acceptance; small sample sizes; and
scarce data on clinical outcomes, such as morbidity, length
of stay, or adverse events. We rated this level of evidence as
low. Most of these studies were fair quality and were eval-
uated in academic medical settings with established health
IT infrastructures and experienced EHR users, which may
limit the generalizability of the findings.
Provider satisfaction with CDSSs was examined in 19

studies (14, 23-25, 37, 43, 80, 86, 105, 109, 112, 119,
127, 128, 141, 151-153, 155, 156, 158, 165). Topics
addressed included diagnosis (43, 112, 151, 152), pharma-
cotherapy (25, 80, 119, 127, 128, 141, 153, 165), chronic
disease management (14, 43, 112, 141), laboratory test
ordering (37, 80, 105), preventive care (37, 43, 80), and
initiating discussions with patients (109).
Comparators included usual care or no CDSS and di-

rect comparison with the same CDSS with additional fea-
tures. Seven CDSSs required a mandatory response (37,
119, 141, 151, 152, 155) or justification (105, 112) for
not adhering to the recommendation. Limitations included
the narrow assessment of the role of provider satisfaction
with CDSSs on patient-specific outcomes and small sample
sizes of clinicians.
Twelve studies demonstrated provider satisfaction

with CDSSs (25, 37, 80, 109, 112, 119, 127, 128, 141,
155, 156, 158, 165); 4 showed a significant effect of satis-
faction among intervention providers compared with con-
trol providers (109, 112, 155, 165). Provider dissatisfac-
tion with CDSSs was also reported in 6 studies (14, 43, 86,
105, 151-153). We rated this level of evidence as moder-
ate. Most studies were good quality and evaluated CDSSs
integrated into CPOE or EHR systems in multiple inter-
ventions outside of environments with an established and
robust health IT. However, most CDSSs were locally de-
veloped and implemented in the ambulatory setting.
3 July 2012 |Annals of Internal Medicine | Volume 157 ® Number 1|37
Seventeen studies examined provider use of CDSSs by

using such metrics as the number of times the CDSS was
accessed by the clinician or provided a recommendation to
the clinician (51, 71, 80, 86, 107, 110, 117, 119, 123,
138, 142, 145, 156, 165, 168-172). Objectives included
diagnosis (71, 123, 138), pharmacotherapy (80, 110, 117,
119, 123, 138, 142, 145, 165), chronic disease manage-
ment (51, 110, 123, 168-172), laboratory test ordering
(71, 80, 107, 110), preventive care (71, 80), and additional
clinical tasks (86, 110, 156).

comparison with the same CDSS with additional features,
or comparison of the same CDSS for different conditions.
Limitations included sparse data demonstrating how pro-
vider use translated into more appropriate patient care and
small sample sizes of clinicians. We rated this level of evi-
dence as low.
Among the 12 studies (80, 107, 117, 119, 123, 138,

145, 168-172) that provided statistical data about pro-
vider use, 8 (80, 110, 119, 123, 145, 165, 168—170) doc-
umented low use (<50% of the clinician’s time or of pa-
tient visits) or that less than 50% of clinicians used the
CDSS or received alerts to guide therapeutic action. Most
of these studies were fair quality and evaluated locally de-
veloped interventions in multiple community and ambula-
tory settings. Additional results are available from the tech-
nical report at www.effectivehealthcare.ahrq.gov.
Discussion
Our systematic review investigated the continuum of

information support for clinical care, including traditional
CDSSs, as well as information retrieval systems and knowl-
edge resources developed for access at the point of care.
Studies were primarily conducted outside of institutions
with an established health IT infrastructure. Most inter-
ventions targeted specific medical conditions and were
evaluated in single settings.
Clinical decision support had a favorable effect on pre-

scribing treatments, facilitating preventive care services,
and ordering clinical studies across diverse venues and sys-
tems. This finding contrasts with that of another review
(2), which showed that most reports of successful CDSS
implementation were based on locally developed systems at
4 sites.
Evidence demonstrating positive effects of CDSSs on

clinical and economic outcomes remains surprisingly
sparse, although this could be because of the relative difh-
culty of implementing randomized, controlled trials in real
clinical settings, as well as the logistics of measuring the
direct clinical effect of CDSSs. Evidence was also limited
in showing an effect of CDSSs on clinical workload and
efficiency. Furthermore, available evidence is insufficient to
draw conclusions about the potential negative effect of im-
plementing decision-support tools, which is necessary to
truly fulfill the goal of evaluating these interventions and to

better address implementation challenges (173).
Our findings are important in light of the increasing

political interest and financial investment of the U.S. gov-
ernment in resources for health IT. Meaningful use of
CDSSs needs to be objectively informed about the role
that they can and should play in reshaping health care
delivery. Further understanding is increasingly important
to optimally define their role in the context of meaningful
use for EHRs.
Our systematic review has several limitations. The het-

erogeneity of the studies limited general observations about
CDSSs. We minimized this limitation in our meta-analyses
by including studies that assessed the same outcome in the
same manner. Although this investigation was a compre-
hensive review of randomized, controlled trials that pro-
vided the best evidence on CDSS effectiveness, these stud-
ies may provide less information about issues related to
CDSS implementation, effect on workflow, and factors af-
fecting usability.
Most studies (76%) evaluated the effectiveness of the
intervention by using usual care rather than a direct com-
parator, which may contribute to more positive results.
Finally, we acknowledge the possibility of selective report-
ing or publication bias. However, a recent review by
Buntin and colleagues (173) found that 62% of included
studies on health IT reported positive effects where the
technology was associated with improvement in 1 or more
aspects of care (173). Formal assessment by using funnel
plots found no consistent bias for most outcomes, except
for ordering or completing of clinical studies, where there
was a strong suggestion of publication bias.
Significant research is still required to promote wide-

spread use of CDSSs and to augment their clinical effec-
tiveness. Future studies should investigate how to expand
CDSS content to accommodate multiple comorbid condi-
tions simultaneously and to determine which members of
the care team should receive clinical decision support, what
effect CDSSs have on clinical and economic outcomes, and
how CDSSs can be most effectively integrated into work-
flow and deployed across diverse settings. Further work is
also needed to understand how CDSSs can aid in the
transformation of care delivery models, such as accountable
care organizations and patient-centered medical homes;
how to incorporate CDSSs into workflow tools, such as
medical registries and provider—provider messaging capa-
bilities; and how to integrate CDSSs with workflow-
oriented quality improvement programs.
Promoting extensive use of CDSSs will require a better

definition of the clinical decision-support infrastructure.
Such infrastructure could include consistent underlying
frameworks for describing CDSSs, such as the “Clinical
Decision Support Five Rights” (174), to aid in the aggre-
gation and synthesis of results; development and evaluation
of models for porting CDSSs across settings; and improved
identification of characteristics of the environment and
www.annals.org
workflow into which a CDSS is deployed, as well as char-

acteristics of the intended users.
In summary, evidence demonstrated the efficacy of

CDSSs on health care process outcomes across diverse set-
tings by using both commercially and locally developed
systems, but data showing an effect on clinical and eco-
nomic outcomes were sparse. Broad penetration of clinical
decision-support tools will require aggressively secking a
better understanding of what the right information is and
when and how it should be delivered to the right person,
and a critical examination of the unintended consequences
of CDSS implementation.
From Duke FEvidence-based Practice Center, Duke Clinical Research

Institute, Duke University School of Medicine, Durham, North Caro-
lina, and School of Medicine, University of Utah, Salt Lake City, Utah.
Disclaimer: The authors of this report are responsible for its content.
Statements in the report should not be construed as endorsements by the
Agency for Healthcare Research and Quality or the U.S. Department of
Health and Human Services.
Acknowledgment: The authors thank Connie Schardt, MSLS, for help
with the literature search and retrieval.
Grant Support: This project was funded under contract 290-2007-

10066-1 from the Agency for Healthcare Research and Quality, U.S.
Department of Health and Human Services.
Potential Conflicts of Interest: Disclosures can be viewed at www

.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M11
-1215.
Requests for Single Reprints: Gillian D. Sanders, PhD, Evidence-based

Practice Center, Director, Duke Clinical Research Institute, 2400 Pratt
Street, Durham, NC 27705; e-mail, [email protected].
Current author addresses and author contributions are available at www

.annals.org.
References
1. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical prac-
tice using clinical decision support systems: a systematic review of trials to
identify
features critical to success. BMJ. 2005;330:765. [PMID: 15767266]
2. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Sys-

tematic review: impact of health information technology on quality, efficiency,
and costs of medical care. Ann Intern Med. 2006;144:742-52. [PMID:
16702590]
3. Bryan C, Boren SA. The use and effectiveness of electronic clinical decision
support tools in the ambulatory/primary care setting: a systematic review of the
literature. Inform Prim Care. 2008;16:79-91. [PMID: 18713524]
4. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ,

Beyene ], et al. Effects of computerized clinical decision support systems on
practitioner performance and patient outcomes: a systematic review. JAMA.
2005;293:1223-38. [PMID: 15755945]
5. Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, et al.

Developing and implementing clinical practice guidelines. Qual Health Care.
1995;4:55-64. [PMID: 10142039]
6. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based

clinical decision support systems on physician performance and patient outcomes:
a systematic review. JAMA. 1998;280:1339-46. [PMID: 9794315]
www.annals.org

7. Sintchenko V, Magrabi F, Tipper S. Are we measuring the right end-points?

Variables that affect the impact of computerised decision support on patient
outcomes: a systematic review. Med Inform Internet Med. 2007;32:225-40.
[PMID: 17701828]
8. Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J.

Effect of point-of-care computer reminders on physician behaviour: a systematic
review. Can Med Assoc J. 2010;182:E216-25.
9. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order

entry and clinical decision support systems on medication safety: a systematic
review. Arch Intern Med. 2003;163:1409-16. [PMID: 12824090]
10. Agency for Healthcare Research and Quality. Methods Guide for Effective-
ness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Health-
care Research and Quality; 2011. Accessed at www.effectivehealthcare.ahrq.gov
/index.cfm/search-for-guides-reviews-and-reports/? pageaction =displayproduct&
productid=318 on 24 January 2012.
11. Owens DK, Lohr KN, Atkins D, Treadwell JR, Reston JT, Bass EB, et al.
AHRQ series paper 5: grading the strength of a body of evidence when compar-
ing medical interventions—agency for healthcare research and quality and the
effective health-care program. ] Clin Epidemiol. 2010;63:513-23. [PMID:
195955771
12. Atkins D, Chang SM, Gartlehner G, Buckley DI, Whitlock EP, Berliner E,
et al. Assessing applicability when comparing medical interventions: AHRQ
and the Effective Health Care Program. ] Clin Epidemiol. 2011;64:1198-207.
[PMID: 21463926]
13. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials.

1986;7:177-88. [PMID: 3802833]
14. McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas
GE. Lessons from a randomized controlled trial designed to evaluate computer
decision support software to improve the management of asthma. Med Inform
Internet Med. 2001;26:191-201. [PMID: 11706929]
15. Cavalcanti AB, Silva E, Pereira AJ, Caldeira-Filho M, Almeida FP, West-
phal GA, et al. A randomized controlled trial comparing a computer-assisted
insulin infusion protocol with a strict and a conventional protocol for glucose
control in critically ill patients. J Crit Care. 2009;24:371-8. [PMID: 19592202]
16. Kline JA, Zeitouni RA, Hernandez-Nino J, Jones AE. Randomized trial of
computerized quantitative pretest probability in low-risk chest pain patients: ef-
fect on safety and resource use. Ann Emerg Med. 2009;53:727-35.e1. [PMID:
19135281]
17. Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B,

et al. Electronic alerts to prevent venous thromboembolism among hospitalized
patients. N Engl ] Med. 2005;352:969-77. [PMID: 15758007]
18. Zanetti G, Flanagan HL Jr, Cohn LH, Giardina R, Platt R. Improvement

of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by auto-
mated alerts in the operating room. Infect Control Hosp Epidemiol. 2003;24:
13-6. [PMID: 12558230]
19. McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger
M, et al. Reminders to physicians from an introspective computer medical re-
cord. A two-year randomized trial. Ann Intern Med. 1984;100:130-8. [PMID:
6691639]
20. Roumie CL, Elasy TA, Greevy R, Griffin MR, Liu X, Stone WJ, et al.
Improving blood pressure control through provider education, provider alerts,
and patient education: a cluster randomized trial. Ann Intern Med. 2006;145:
165-75. [PMID: 16880458]
21. Paul M, Andreassen S, Tacconelli E, Nielsen AD, Almanasreh N, Frank U,

et al; TREAT Study Group. Improving empirical antibiotic treatment using
TREAT, a computerized decision support system: cluster randomized trial. J
Antimicrob Chemother. 2006;58:1238-45. [PMID: 16998208]
22. Ansari M, Shlipak MG, Heidenreich PA, Van Ostaeyen D, Pohl EC,
Browner WS, et al. Improving guideline adherence: a randomized trial evaluating
strategies to increase beta-blocker use in heart failure. Circulation. 2003;107:
2799-804. [PMID: 12756157]
23. Graumlich JF, Novotny NL, Nace GS, Aldag JC. Patient and physician
perceptions after software-assisted hospital discharge: cluster randomized trial. J
Hosp Med. 2009;4:356-63. [PMID: 19621342]
24. Graumlich JF, Novotny NL, Stephen Nace G, Kaushal H, Ibrahim-Ali W,

Theivanayagam S, et al. Patient readmissions, emergency visits, and adverse
events after software-assisted discharge from hospital: cluster randomized trial. ]
Hosp Med. 2009;4:E11-9. [PMID: 19479782]
25. Heidenreich PA, Gholami P, Sahay A, Massie B, Goldstein MK. Clinical
reminders attached to echocardiography reports of patients with reduced left
3 July 2012 | Annals of Internal Medicine | Volume 157 ¢ Number 1|39
ventricular ejection fraction increase use of beta-blockers: a randomized trial.

Circulation. 2007;115:2829-34. [PMID: 17515459]
26. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert
GJ, et al. Can computer-generated evidence-based care suggestions enhance
evidence-based management of asthma and chronic obstructive pulmonary dis-
ease? A randomized, controlled trial. Health Serv Res. 2005;40:477-97. [PMID:
15762903]
27. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert
GJ, et al. Effects of computerized guidelines for managing heart disease in pri-
mary care. ] Gen Intern Med. 2003;18:967-76. [PMID: 14687254]
28. Gilutz H, Novack L, Shvartzman P, Zelingher J, Bonneh DY, Henkin Y,

et al. Computerized community cholesterol control (4C): meeting the challenge
of secondary preventon. Ist Med Assoc J. 2009;11:23-9. [PMID: 19344008]
29. Holt TA, Thorogood M, Griffiths F, Munday S. Protocol for the ‘e-Nudge
trial’: a randomised controlled trial of electronic feedback to reduce the cardio-
vascular risk of individuals in general practice [[SRCTN64828380]. Trials. 20065
7:11. [PMID: 16646967]
30. Holt TA, Thorogood M, Griffiths F, Munday S, Friede T, Stables D.

Automated electronic reminders to facilitate primary cardiovascular disease pre-
vention: randomised controlled trial. Br ] Gen Pract. 2010;60:e137-43. [PMID:
20353659]
31. Khan S, Maclean CD, Littenberg B. The effect of the Vermont Diabetes
Information System on inpatient and emergency room use: results from a ran-
domized trial. Health Outcomes Res Med. 2010;1:e61-e66. [PMID: 20975923]
32. Maclean CD, Gagnon M, Callas P, Littenberg B. The Vermont diabetes
information system: a cluster randomized trial of a population based decision
support system. ] Gen Intern Med. 2009;24:1303-10. [PMID: 19862578]
33. Brier ME, Gaweda AE, Dailey A, Aronoff GR, Jacobs AA. Randomized trial
of model predictive control for improved anemia management. Clin ] Am Soc
Nephrol. 201055:814-20. [PMID: 20185598]
34. Hamilton E, Platt R, Gauthier R, McNamara H, Miner L, Rothenberg S,

et al. The effect of computer-assisted evaluation of labor on cesarean rates. ]
Healthc Qual. 2004;26:37-44. [PMID: 14763319]
35. McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for
influenza vaccination on morbidity during influenza epidemics. MD Comput.
1992;9:304-12. [PMID: 1522792]
36. Murray MD, Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE,
et al. Failure of computerized treatment suggestions to improve health outcomes
of outpatients with uncomplicated hypertension: results of a randomized con-
trolled trial. Pharmacotherapy. 2004;24:324-37. [PMID: 15040645]
37. Sequist TD, Zaslavsky AM, Marshall R, Fletcher RH, Ayanian JZ. Patient
and physician reminders to promote colorectal cancer screening: a randomized
controlled trial. Arch Intern Med. 2009;169:364-71. [PMID: 19237720]
38. Subramanian U, Fihn SD, Weinberger M, Plue L, Smith FE, Udris EM,
et al. A controlled trial of including symptom data in computer-based care sug-
gestions for managing patients with chronic heart failure. Am ] Med. 2004;116:
375-84. [PMID: 15006586]
39. Kuperman GJ, Teich JM, Tanasijevic M], Ma'Luf N, Rittenberg E, Jha A,
et al. Improving response to critical laboratory results with automation: results
of
a randomized controlled trial. ] Am Med Inform Assoc. 1999;6:512-22. [PMID:
10579608]
40. McGregor JC, Weekes E, Forrest GN, Standiford HC, Perencevich EN,
Furuno JP, et al. Impact of a computerized clinical decision support system on
reducing inappropriate antimicrobial use: a randomized controlled trial. ] Am
Med Inform Assoc. 2006;13:378-84. [PMID: 16622162]
41. Fihn SD, McDonell MB, Vermes D, Henikoff JG, Martin DC, Callahan
CM, et al. A computerized intervention to improve timing of outpatient follow-
up: a multicenter randomized trial in patients treated with warfarin. National
Consortium of Anticoagulation Clinics. ] Gen Intern Med. 1994;9:131-9.
[PMID: 8195911]
42. Gurwitz JH, Field TS, Rochon P, Judge ], Harrold LR, Bell CM, et al.
Effect of computerized provider order entry with clinical decision support on
adverse drug events in the long-term care setting. ] Am Geriatr Soc. 2008;56:
2225-33. [PMID: 19093922]
43. Apkon M, Mattera JA, Lin Z, Herrin ], Bradley EH, Carbone M, et al. A
randomized outpatient trial of a decision-support information technology tool.
Arch Intern Med. 2005;165:2388-94. [PMID: 16287768]
44. Bertoni AG, Bonds DE, Chen H, Hogan P, Crago L, Rosenberger E, et al.
Impact of a multifaceted intervention on cholesterol management in primary care
practices: guideline adherence for heart health randomized trial. Arch Intern Med.
2009;169:678-86. [PMID: 19364997]
45. Burack RC, Gimotty PA, George J, McBride S, Moncrease A, Simon MS,
et al. How reminders given to patients and physicians affected pap smear use in
a health maintenance organization: results of a randomized controlled trial. Can-
cer. 1998;82:2391-400. [PMID: 9635532]
46. Burack RC, Gimotty PA, Simon M, Moncrease A, Dews P. The effect of
adding Pap smear information to a mammography reminder system in an HMO:
results of randomized controlled trial. Prev Med. 2003;36:547-54. [PMID:
12689799]
47. Cannon DS, Allen SN. A comparison of the effects of computer and manual
reminders on compliance with a mental health clinical practice guideline. ] Am
48. Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R,
et al. Improving residents’ compliance with standards of ambulatory care: results
from the VA Cooperative Study on Computerized Reminders. JAMA. 2000;284:
1411-6. [PMID: 10989404]
49. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald
CJ. A computerized reminder system to increase the use of preventive care for
hospitalized patients. N Engl ] Med. 2001;345:965-70. [PMID: 11575289]
50. Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. Inpatient
computer-based standing orders vs physician reminders to increase influenza and
pneumococcal vaccination rates: a randomized trial. JAMA. 2004;292:2366-71.
[PMID: 15547164]
51. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al.

Effect of computerised evidence based guidelines on management of asthma and
angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;
325:941. [PMID: 12399345]
52. Frank O, Litt J, Beilby J. Opportunistic electronic reminders. Improving

performance of preventive care in general practice. Aust Fam Physician. 2004;33:
87-90. [PMID: 14988972]
53. Fretheim A, Aaserud M, Oxman AD. Rational prescribing in primary care

(RaPP): economic evaluation of an intervention to improve professional practice.
PLoS Med. 2006;3:€216. [PMID: 16737349]
54. Fretheim A, Oxman AD, Havelsrud K, Treweek S, Kristoffersen DT,

Bjorndal A. Rational prescribing in primary care (RaPP): a cluster randomized
trial of a tailored intervention. PLoS Med. 2006;3:e134. [PMID: 167373406]
55. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians
to respond to computerized reminders improves their compliance with preventive
care protocols. ] Gen Intern Med. 1993;8:311-7. [PMID: 8320575]
56. McDowell I, Newell C, Rosser W. Comparison of three methods of recalling

patients for influenza vaccination. CMAJ. 1986;135:991-7. [PMID: 3093045]
57. McDowell I, Newell C, Rosser W. Computerized reminders to encourage
cervical screening in family practice. ] Fam Pract. 1989;28:420-4. [PMID:
24953371
58. Overhage JM, Tierney WM, McDonald CJ. Computer reminders to imple-
ment preventive care guidelines for hospitalized patients. Arch Intern Med. 1996;
156:1551-6. [PMID: 8687263]
59. Price M. Can hand-held computers improve adherence to guidelines? A

(Palm) Pilot study of family doctors in British Columbia. Can Fam Physician.
2005;51:1506-7. [PMID: 16926943]
60. Taylor V, Thompson B, Lessler D, Yasui Y, Montano D, Johnson KM,

et al. A clinic-based mammography intervention targeting inner-city women. J
Gen Intern Med. 1999;14:104-11. [PMID: 10051781]
61. Unrod M, Smith M, Spring B, DePue J, Redd W, Winkel G. Randomized

controlled trial of a computer-based, tailored intervention to increase smoking
cessation counseling by primary care physicians. ] Gen Intern Med. 2007;22:478-
84. [PMID: 17372796]
62. Dykes PC, Carroll DL, Hurley A, Lipsitz S, Benoit A, Chang F, et al. Fall
prevention in acute care hospitals: a randomized trial. JAMA. 2010;304:1912-8.
[PMID: 21045097]
63. Chambers CV, Balaban DJ, Carlson BL, Ungemack JA, Grasberger DM.
Microcomputer-generated reminders. Improving the compliance of primary care
physicians with mammography screening guidelines. ] Fam Pract. 1989;29:273-
80. [PMID: 2769192]
64. Burack RC, Gimotty PA. Promoting screening mammography in inner-city

settings. The sustained effectiveness of computerized reminders in a randomized
controlled trial. Med Care. 1997;35:921-31. [PMID: 9298081]
65. Burack RC, Gimotty PA, George ], Stengle W, Warbasse L, Moncrease A.

Promoting screening mammography in inner-city settings: a randomized con-
www.annals.org
trolled trial of computerized reminders as a component of a program to facilitate

mammography. Med Care. 1994;32:609-24. [PMID: 8189778]
66. Fiks AG, Hunter KF, Localio AR, Grundmeier RW, Bryant-Stephens T,
Luberti AA, et al. Impact of electronic health record-based alerts on influenza
vaccination for children with asthma. Pediatrics. 2009;124:159-69. [PMID:
19564296]
67. Flanagan JR, Doebbeling BN, Dawson J, Beekmann S. Randomized study

of online vaccine reminders in adult primary care. Proc AMIA Symp. 1999:
755-9. [PMID: 10566461]
68. Fordham D, McPhee SJ, Bird JA, Rodnick JE, Detmer WM. The Cancer
Prevention Reminder System. MD Comput. 1990;7:289-95. [PMID: 2243544]
69. McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening.
A randomized, controlled trial of three interventions. Arch Intern Med. 1989;
149:1866-72. [PMID: 2764657]
70. Gill JM, Chen YX, Glutting JJ, Diamond JJ, Lieberman MI. Impact of
decision support in electronic medical records on lipid management in primary
care. Popul Health Manag. 2009;12:221-6. [PMID: 19848563]
71. Hobbs FD, Delaney BC, Carson A, Kenkre JE. A prospective controlled
trial of computerized decision support for lipid management in primary care.
Fam Pract. 1996;13:133-7. [PMID: 8732323]
72. Holbrook A, Thabane L, Keshavjee K, Dolovich L, Bernstein B, Chan D,

et al; COMPETE II Investigators. Individualized electronic decision support
and reminders to improve diabetes care in the community: COMPETE II ran-
domized trial. CMAJ. 2009;181:37-44. [PMID: 19581618]
73. Kenealy T, Arroll B, Petrie KJ. Patients and computers as reminders to

screen for diabetes in family practice. Randomized-controlled trial. ] Gen Intern
Med. 2005;20:916-21. [PMID: 16191138]
74. Lobach DF, Hammond WE. Development and evaluation of a Computer-

Assisted Management Protocol (CAMP): improved compliance with care guide-
lines for diabetes mellitus. Proc Annu Symp Comput Appl Med Care. 1994:787-
91. [PMID: 7950032]
75. Ornstein SM, Garr DR, Jenkins RG, Rust PF, Amon A. Computer-
generated physician and patient reminders. Tools to improve population adher-
ence to selected preventive services. ] Fam Pract. 1991;32:82-90. [PMID:
1985140]
76. Peterson KA, Radosevich DM, O’Connor PJ, Nyman JA, Prineas R],
Smith SA, et al. Improving diabetes care in practice: findings from the TRANS-
LATE trial. Diabetes Care. 2008;31:2238-43. [PMID: 18809622]
77. Reeve JF, Tenni PC, Peterson GM. An electronic prompt in dispensing
software to promote clinical interventions by community pharmacists: a random-
ized controlled trial. Br ] Clin Pharmacol. 2008;65:377-85. [PMID: 17764471]
78. Rosser WW, Hutchison BG, McDowell I, Newell C. Use of reminders to
increase compliance with tetanus booster vaccination. CMA]J. 1992;146:911-7.
[PMID: 1544078
79. Rosser WW, McDowell I, Newell C. Use of reminders for preventive pro-
cedures in family medicine. CMA]J. 1991;145:807-14. [PMID: 1913409]
80. Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Spetling M,
etal. A randomized trial of electronic clinical reminders to improve quality of
care
for diabetes and coronary artery disease. ] Am Med Inform Assoc. 2005;12:
431-7. [PMID: 15802479]
81. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician per-
formance versus immediate reminders to perform preventive care. Effects on
physician compliance. Med Care. 1986;24:659-66. [PMID: 3736141]
82. van Wyk JT, van Wijk MA, Sturkenboom MC, Mosseveld M, Moorman
PW, van der Lei ]J. Electronic alerts versus on-demand decision support to im-
prove dyslipidemia treatment: a cluster randomized controlled trial. Circulation.
2008;117:371-8. [PMID: 18172036]
83. Bates DW, Kuperman GJ, Rittenberg E, Teich JM, Fiskio ], Ma'luf N,
et al. A randomized trial of a computer-based intervention to reduce utilization of
redundant laboratory tests. Am J Med. 1999;106:144-50. [PMID: 10230742]
84. Bell LM, Grundmeier R, Localio R, Zorc ], Fiks AG, Zhang X, et al.
Electronic health record-based decision support to improve asthma care: a duster-
randomized trial. Pediatrics. 2010;125:¢770-7. [PMID: 20231191]
85. Downs M, Tumer S, Bryans M, Wilcock ], Keady J, Levin E, et al.

Effectiveness of educational interventions in improving detection and manage-
ment of dementia in primary care: cluster randomised controlled study. BM]J.
2006;332:692-6. [PMID: 16565124]
86. Emery ], Morris H, Goodchild R, Fanshawe T, Prevost AT, Bobrow M,

et al. The GRAIDS Trial: a cluster randomised controlled trial of computer
www.annals.org
decision support for the management of familial cancer risk in primary care. Br ]
Cancer. 2007;97:486-93. [PMID: 17700548]
87. Feldstein AC, Smith DH, Perrin N, Yang X, Rix M, Raebel MA, et al.
Improved therapeutic monitoring with several interventions: a randomized trial.
Arch Intern Med. 2006;166:1848-54. [PMID: 17000941]
88. Flottorp S, Oxman AD, Havelsrud K, Treweek S, Herrin J. Cluster ran-

domised controlled trial of tailored interventions to improve the management of
urinary tract infections in women and sore throat. BMJ. 2002;325:367. [PMID:
12183309]
89. Greiver M, Drummond N, White D, Weshler J, Moineddin R; North

Toronto Primary Care Research Network (Nortren). Angina on the Palm: ran-
domized controlled pilot trial of Palm PDA software for referrals for cardiac
testing. Can Fam Physician. 2005;51:382-3. [PMID: 16926933]
90. Harpole LH, Khorasani R, Fiskio J, Kuperman GJ, Bates DW. Automated
evidence-based critiquing of orders for abdominal radiographs: impact on utiliza-
tion and appropriateness. ] Am Med Inform Assoc. 1997;4:511-21. [PMID:
9391938]
91. Lee NJ, Chen ES, Currie LM, Donovan M, Hall EK, Jia H, et al. The effect
of a mobile clinical decision support system on the diagnosis of obesity and
overweight in acute and primary care encounters. ANS Adv Nurs Sci. 2009;32:
211-21. [PMID: 19707090]
92. Lo HG, Matheny ME, Seger DL, Bates DW, Gandhi TK. Impact of
non-interruptive medication laboratory monitoring alerts in ambulatory care. ]
Am Med Inform Assoc. 2009;16:66-71. [PMID: 18952945]
93. Matheny ME, Sequist TD, Seger AC, Fiskio JM, Sperling M, Bugbee D,
et al. A randomized trial of electronic clinical reminders to improve medication
laboratory monitoring. ] Am Med Inform Assoc. 2008;15:424-9. [PMID:
18436905]
94. McDonald CJ. Use of a computer to detect and respond to clinical events: its
effect on clinician behavior. Ann Intern Med. 1976;84:162-7. [PMID: 1252043]
95. McDowell I, Newell C, Rosser W. A randomized trial of computerized
reminders for blood pressure screening in primary care. Med Care. 1989;27:297-
305. [PMID: 2494397]
96. Palen TE, Raebel M, Lyons E, Magid DM. Evaluation of laboratory mon-
itoring alerts within a computerized physician order entry system for medication
orders. Am ] Manag Care. 2006;12:389-95. [PMID: 16834525]
97. Palen TE, Price DW, Snyder AJ, Shetterly SM. Computerized alert reduced
D-dimer testing in the elderly. Am ] Manag Care. 2010;16:¢267-75. [PMID:
21087072]
98. Player MS, Gill JM, Mainous AG 3rd, Everett CJ, Koopman R]J, Diamond
JJ, et al. An electronic medical record-based intervention to improve quality of
care for gastro-esophageal reflux disease (GERD) and atypical presentations of
GERD. Qual Prim Care. 2010;18:223-9. [PMID: 20836938]
99. Raebel MA, Lyons EE, Chester EA, Bodily MA, Kelleher JA, Long CL,
et al. Improving laboratory monitoring at initiation of drug therapy in ambula-
tory care: a randomized trial. Arch Intern Med. 2005;165:2395-401. [PMID:
16287769]
100. Raebel MA, Chester EA, Newsom EE, Lyons EE, Kelleher JA, Long C,
et al. Randomized trial to improve laboratory safety monitoring of ongoing drug
therapy in ambulatory patients. Pharmacotherapy. 2006;26:619-26. [PMID:
16637791]
101. Roukema J, Steyerberg EW, van der Lei J, Moll HA. Randomized trial of
a clinical decision support system: impact on the management of children with
fever without apparent source. ] Am Med Inform Assoc. 2008;15:107-13.
[PMID: 17947627]
102. Roy PM, Durieux P, Gillaizeau F, Legall C, Armand-Perroux A, Martino

L, et al. A computerized handheld decision-support system to improve pulmo-
nary embolism diagnosis: a randomized trial. Ann Intern Med. 2009;151:677-86.
[PMID: 19920268]
103. Schriefer SP, Landis SE, Turbow D]J, Patch SC. Effect of a computerized
body mass index prompt on diagnosis and treatment of adult obesity. Fam Med.
2009;41:502-7. [PMID: 19582630]
104. Stiell IG, Clement CM, Grimshaw J, Brison R], Rowe BH, Schull M],
et al. Implementation of the Canadian C-Spine Rule: prospective 12 centre clus-
ter randomised trial. BM]. 2009;339:b4146. [PMID: 19875425]
105. Sundaram V, Lazzeroni LC, Douglass LR, Sanders GD, Tempio P,

Owens DK. A randomized trial of computer-based reminders and audit and
feedback to improve HIV screening in a primary care setting. Int ] STD AIDS.
2009;20:527-33. [PMID: 19625582]
3 July 2012 | Annals of Internal Medicine [ Volume 157 ¢ Number 1 |41
106. Tierney WM, McDonald CJ, Martin DK, Rogers MP. Computerized
display of past test results. Effect on outpatient testing. Ann Intern Med. 1987;
107:569-74. [PMID: 3631792]
107. van Wijk MA, van der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH.
Assessment of decision support for blood test ordering in primary care. a random-
ized trial. Ann Intern Med. 2001;134:274-81. [PMID: 11182837]
108. Walker ], Fairley CK, Walker SM, Gurrin LC, Gunn JM, Pirotta MV,
et al. Computer reminders for Chlamydia screening in general practice: a ran-
domized controlled trial. Sex Transm Dis. 2010;37:445-50. [PMID: 20375930]
109. Wilson BJ, Torrance N, Mollison J, Watson MS, Douglas A, Miedzy-
brodzka Z, et al. Cluster randomized trial of a multifaceted primary care
decision-support intervention for inherited breast cancer risk. Fam Pract. 2006;
23:537-44. [PMID: 16787957]
110. Bourgeois FC, Linder J, Johnson SA, Co JP, Fiskio ], Ferris TG. Impact of
a computerized template on antibiotic prescribing for acute respiratory infections
in children and adolescents. Clin Pediatr (Phila). 2010;49:976-83. [PMID:
20724348]
111. Christakis DA, Zimmerman FJ, Wright JA, Garrison MM, Rivara FP,
Davis RL. A randomized controlled trial of point-of-care evidence to improve the
antibiotic prescribing practices for otitis media in children. Pediatrics.
2001;107:
E15. [PMID: 11158489]
112. Co JP, Johnson SA, Poon EG, Fiskio J, Rao SR, Van Cleave ], et al.
Electronic health record decision support and quality of care for children with
ADHD. Pediatrics. 2010;126:239-46. [PMID: 20643719]
113. Cobos A, Vilaseca J, Asenjo C, Pedro-Botet ], Sanchez E, Val A, et al. Cost

effectiveness of a clinical decision support system based on the recommendations
of the European Society of Cardiology and other societies for the management of
hypercholesterolemia: report of a cluster-randomized trial. Disease Management
& Health Outcomes. 2005;13:421-32.
114. Davis RL, Wright ], Chalmers F, Levenson L, Brown JC, Lozano P, et al.
A cluster randomized clinical trial to improve prescribing patterns in ambulatory
pediatrics. PLoS Clin Trials. 2007;2:€25. [PMID: 17525793]
115. Feldstein A, Elmer PJ, Smith DH, Herson M, Orwoll E, Chen C, et al.
Electronic medical record reminder improves osteoporosis management after a
fracture: a randomized, controlled trial. ] Am Geriatr Soc. 2006;54:450-7.
[PMID: 16551312]
116. Field TS, Rochon P, Lee M, Gavendo L, Baril JL, Gurwitz JH. Comput-
erized clinical decision support during medication ordering for long-term care
residents with renal insufficiency. ] Am Med Inform Assoc. 2009;16:480-5.
[PMID: 19390107]
117. Filippi A, Sabatini A, Badioli L, Samani F, Mazzaglia G, Catapano A,

et al. Effects of an automated electronic reminder in changing the antiplatelet
drug-prescribing behavior among Italian general practitioners in diabetic patients:
an intervention trial. Diabetes Care. 2003;26:1497-500. [PMID: 12716811]
118. Fitzmaurice DA, Hobbs FD, Murray ET, Holder RL, Allan TF, Rose PE.
Oral anticoagulation management in primary care with the use of computerized
decision support and near-patient testing: a randomized, controlled trial. Arch
Intern Med. 2000;160:2343-8. [PMID: 10927732]
119. Fortuna R], Zhang F, Ross-Degnan D, Campion FX, Finkelstein JA,

Kotch JB, et al. Reducing the prescribing of heavily marketed medications: a
randomized controlled trial. ] Gen Intern Med. 2009;24:897-903. [PMID:
19475459]
120. Goud R, de Keizer NF, ter Riet G, Wyatt JC, Hasman A, Hellemans IM,
et al. Effect of guideline based computerised decision support on decision making
of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation.
BM]J.
2009;338:b1440. [PMID: 19398471]
121. Hicks LS, Sequist TD, Ayanian JZ, Shaykevich S, Fairchild DG, Orav EJ,
et al. Impact of computerized decision support on blood pressure management
and control: a randomized controlled trial. ] Gen Intern Med. 2008;23:429-41.
[PMID: 18373141]
122. Krall MA, Traunweiser K, Towery W. Effectiveness of an electronic med-

ical record clinical quality alert prepared by off-line data analysis. Stud Health
Technol Inform. 2004;107:135-9. [PMID: 15360790]
123. Linder JA, Rigotti NA, Schneider LI, Kelley JH, Brawarsky P, Haas JS. An
electronic health record-based intervention to improve tobacco treatment in pri-
mary care: a cluster-randomized controlled trial. Arch Intern Med. 2009;169:
781-7. [PMID: 19398690]
124. Locatelli F, Covic A, Macdougall IC, Scherhag A, Wiecek A; ORAMA

Study Group. Effect of computer-assisted European Best Practice Guideline im-
plementation on adherence and target attainment: ORAMA results. ] Nephrol.

2009;22:662-74. [PMID: 19810000]
125. Manotti C, Moia M, Palareti G, Pengo V, Ria L, Dettori AG. Effect of

computer-aided management on the quality of treatment in anticoagulated pa-
tients: a prospective, randomized, multicenter trial of APROAT (Automated
PRogram for Oral Anticoagulant Treatment). Haematologica. 2001;86:1060-70.
[PMID: 11602412]
126. Marco F, Sedano C, Bermidez A, Lépez-Duarte M, Ferndndez-Fontecha

E, Zubizarreta A. A prospective controlled study of a computer-assisted aceno-
coumarol dosage program. Pathophysiol Haemost Thromb. 2003;33:59-63.
[PMID: 14624045]
127. Martens JD, van der Aa A, Panis B, van der Weijden T, Winkens RA,
Severens JL. Design and evaluation of a computer reminder system to improve
prescribing behaviour of GPs. Stud Health Technol Inform. 2006;124:617-23.
[PMID: 17108585]
128. Martens JD, van der Weijden T, Severens JL, de Clercq PA, de Bruijn
DP, Kester AD, et al. The effect of computer reminders on GPs’ prescribing
behaviour: a cluster-randomised trial. Int ] Med Inform. 2007576 Suppl 3:S403-
16. [PMID: 17569575]
129. Montgomery AA, Fahey T, Peters T], Maclntosh C, Sharp DJ. Evaluation
of computer based clinical decision support system and risk chart for manage-
ment of hypertension in primary care: randomised controlled trial. BMJ. 2000;
320:686-90. [PMID: 10710578]
130. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial
of “corollary orders” to prevent errors of omission. ] Am Med Inform Assoc.
1997;4:364-75. [PMID: 9292842]
131. Peterson JF, Rosenbaum BP, Waitman LR, Habermann R, Powers ],

Harrell D, et al. Physicians’ response to guided geriatric dosing: initial results
from a randomized trial. Stud Health Technol Inform. 2007;129:1037-40.
[PMID: 17911873]
132. Phillips LS, Ziemer DC, Doyle JP, Barnes CS, Kolm P, Branch WT, et al.
An endocrinologist-supported intervention aimed at providers improves diabetes
management in a primary care site: improving primary care of African
Americans with diabetes (IPCAAD) 7. Diabetes Care. 2005;28:2352-60.
[PMID: 16186262]
133. Ziemer DC, Doyle JP, Barnes CS, Branch WT Jr, Cook CB, El-Kebbi
IM, et al. An intervention to overcome clinical inertia and improve diabetes
mellitus control in a primary care setting: Improving Primary Care of African
Americans with Diabetes IPCAAD) 8. Arch Intern Med. 2006;166:507-13.
[PMID: 16534030]
134. Raebel MA, Charles J, Dugan ], Carroll NM, Korner EJ, Brand DW,
et al. Randomized trial to improve prescribing safety in ambulatory elderly pa-
tients. ] Am Geriatr Soc. 2007;55:977-85. [PMID: 17608868]
135. Rood E, Bosman RJ, van der Spoel JI, Taylor P, Zandstra DF. Use of a
computerized guideline for glucose regulation in the intensive care unit improved
both guideline adherence and glucose regulation. ] Am Med Inform Assoc. 2005;
12:172-80. [PMID: 15561795]
136. Rossi RA, Every NR. A computerized intervention to decrease the use of
calcium channel blockers in hypertension. ] Gen Intern Med. 1997;12:672-8.
[PMID: 9383135]
137. Rothschild JM, McGurk S, Honour M, Lu L, McClendon AA, Srivastava

P, et al. Assessment of education and computerized decision support interven-
tions for improving transfusion practice. Transfusion. 2007;47:228-39. [PMID:
17302768]
138. Samore MH, Bateman K, Alder SC, Hannah E, Donnelly S, Stoddard GJ,
et al. Clinical decision support and appropriateness of antimicrobial prescribing:
a randomized trial. JAMA. 2005;294:2305-14. [PMID: 16278358]
139. Shojania KG, Yokoe D, Platt R, Fiskio J, Ma'luf N, Bates DW. Reducing
vancomycin use utilizing a computer guideline: results of a randomized controlled
trial. ] Am Med Inform Assoc. 1998;5:554-62. [PMID: 9824802]
140. Simon SR, Smith DH, Feldstein AC, Perrin N, Yang X, Zhou Y, et al.
Computerized prescribing alerts and group academic detailing to reduce the use
of potentially inappropriate medications in older people. ] Am Geriatr Soc. 2006;
54:963-8. [PMID: 16776793]
141. Smith SA, Shah ND, Bryant SC, Christianson TJ, Bjornsen SS, Giesler
PD, et al; Evidens Research Group. Chronic care model and shared care in
diabetes: randomized trial of an electronic decision support system. Mayo Clin
Proc. 2008;83:747-57. [PMID: 18613991]
142. Strom BL, Schinnar R, Aberra F, Bilker W, Hennessy S, Leonard CE,

etal. Unintended effects of a computerized physician order entry nearly hard-stop
www.annals.org
alert to prevent a drug interaction: a randomized controlled trial. Arch Intern

Med. 2010;170:1578-83. [PMID: 20876410]
143. Strom BL, Schinnar R, Bilker W, Hennessy S, Leonard CE, Pifer E.

Randomized clinical trial of a customized electronic alert requiring an affirmative
response compared to a control group receiving a commercial passive CPOE
alert: NSAID—warfarin co-prescribing as a test case. ] Am Med Inform Assoc.
2010;17:411-5. [PMID: 20595308]
144. Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley ], et al. The

medical office of the 21st century (MOXXI): effectiveness of computerized
decision-making support in reducing inappropriate prescribing in primary care.
CMA]J. 2003;169:549-56. [PMID: 12975221]
145. Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, et al.

A randomized trial of the effectiveness of on-demand versus computer-triggered
drug decision support in primary care. ] Am Med Inform Assoc. 2008;15:430-8.
[PMID: 18436904]
146. Tamblyn R, Reidel K, Huang A, Taylor L, Winslade N, Bartlett G, et al.

Increasing the detection and response to adherence problems with cardiovascular
medication in primary care through computerized drug management systems: a
randomized controlled trial. Med Decis Making. 2010;30:176-88. [PMID:
19675319]
147. Terrell KM, Perkins AJ, Dexter PR, Hui SL, Callahan CM, Miller DK.
Computerized decision support to reduce potentially inappropriate prescribing to
older emergency department patients: a randomized, controlled trial. ] Am Geri-
atr Soc. 2009;57:1388-94. [PMID: 19549022]
148. Terrell KM, Perkins AJ, Hui SL, Callahan CM, Dexter PR, Miller DK.
Computerized decision support for medication dosing in renal insufficiency:
a randomized, controlled trial. Ann Emerg Med. 2010;56:623-9. [PMID:
20452703]
149. Vadher BD, Patterson DL, Leaning M. Comparison of oral anticoagulant

control by a nurse-practitioner using a computer decision-support system with
that by clinicians. Clin Lab Haematol. 1997;19:203-7. [PMID: 9352146]
150. Vadher B, Patterson DL, Leaning M. Evaluation of a decision support

system for initiation and control of oral anticoagulation in a randomised trial.
BM]J. 1997;314:1252-6. [PMID: 9154031]
151. Vissers MC, Biert J, van der Linden CJ, Hasman A. Effects of a supportive
protocol processing system (ProtoVIEW) on clinical behaviour of residents in the
accident and emergency department. Comput Methods Programs Biomed. 1996;
49:177-84. [PMID: 8735024]
152. Vissers MC, Hasman A, van der Linden CJ. Protocol processing system
(ProtoVIEW) to support residents at the emergency ward. Comput Methods
Programs Biomed. 1995;48:53-8. [PMID: 8846712]
153. Weir CJ, Lees KR, MacWalter RS, Muir KW, Wallesch CW, McLelland
EV, etal; PRISM Study Group. Cluster-randomized, controlled trial of computer-
based decision support for selecting long-term anti-thrombotic therapy after
acute ischaemic stroke. QJM. 2003;96:143-53. [PMID: 12589012]
154. White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a
computerized medical decision-making process to the problem of digoxin intox-
ication. ] Am Coll Cardiol. 1984;4:571-6. [PMID: 6381570]
155. Alper BS, White DS, Ge B. Physicians answer more clinical questions and
change clinical decisions more often with synthesized evidence: a randomized trial
in primary care. Ann Fam Med. 2005;3:507-13. [PMID: 16338914]
156. Del Fiol G, Haug PJ, Cimino JJ, Narus SP, Notlin C, Mitchell JA.
Effectiveness of topic-specific infobuttons: a randomized controlled trial. ] Am
157. Etchells E, Adhikari NK, Cheung C, Fowler R, Kiss A, Quan S, et al.

Real-time clinical alerting: effect of an automated paging system on response time
to critical laboratory values—a randomised controlled trial. Qual Saf Health
Care. 2010;19:99-102. [PMID: 20351157]
158. Bird JA, McPhee SJ, Jenkins C, Fordham D. Three strategies to promote
cancer screening. How feasible is wide-scale implementation? Med Care. 1990;
28:1005-12. [PMID: 2250488]
www.annals.org
159. Frame PS, Zimmer JG, Werth PL, Hall W], Eberly SW. Computer-based
vs manual health maintenance tracking. A controlled trial. Arch Fam Med. 1994;
3:581-8. [PMID: 7921293]
160. Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions
of abnormal test results. Effects on outpatient testing. JAMA. 1988;259:1194-8.
[PMID: 3339821]
161. Cleveringa FG, Gorter KJ, van den Donk M, Rutten GE. Combined task
delegation, computerized decision support, and feedback improve cardiovascular
risk for type 2 diabetic patients: a cluster randomized trial in primary care. Dia-
betes Care. 2008;31:2273-5. [PMID: 18796619]
162. Cleveringa FG, Welsing PM, van den Donk M, Gorter KJ, Niessen LW,
Rutten GE, et al. Cost-effectiveness of the diabetes care protocol, a multifaceted
computerized decision support diabetes management intervention that reduces
cardiovascular risk. Diabetes Care. 2010;33:258-63. [PMID: 19933991]
163. Smith DH, Feldstein AC, Perrin NA, Yang X, Rix MM, Raebel MA, et al.
Improving laboratory monitoring of medications: an economic analysis alongside
a clinical trial. Am ] Manag Care. 2009;15:281-9. [PMID: 19435396]
164. Judge J, Field TS, DeFlorio M, Laprino J, Auger J, Rochon P, et al.

Prescribers’ responses to alerts during medication ordering in the long term care
setting. ] Am Med Inform Assoc. 2006;13:385-90. [PMID: 16622171]
165. Maviglia SM, Yoon CS, Bates DW, Kuperman G. KnowledgeLink: impact
of context-sensitive information retrieval on clinicians’ information needs. ] Am
166. Rollman BL, Hanusa BH, Gilbert T, Lowe HJ, Kapoor WN, Schulberg
HC. The electronic medical record. A randomized trial of its impact on primary
care physicians’ initial management of major depression [corrected]. Arch Intern
Med. 2001;161:189-97. [PMID: 11176732]
167. McLaughlin D, Hayes JR, Kelleher K. Office-based interventions for rec-

ognizing abnormal pediatric blood pressures. Clin Pediatr (Phila). 2010;49:355-
62.
168. Hetlevik I, Holmen J, Kriiger O. Implementing clinical guidelines in the

treatment of hypertension in general practice. Evaluation of patient outcome
related to implementation of a computer-based clinical decision support system.
Scand ] Prim Health Care. 1999;17:35-40. [PMID: 10229991]
169. Hetlevik I, Holmen J, Kriiger O, Kristensen P, Iversen H. Implementing

clinical guidelines in the treatment of hypertension in general practice. Blood
Press. 1998;7:270-6. [PMID: 10321438]
170. Hetlevik I, Holmen J, Kriiger O, Kristensen P, Iversen H, Furuseth K.

Implementing clinical guidelines in the treatment of diabetes mellitus in general
practice. Evaluation of effort, process, and patient outcome related to implemen-
tation of a computer-based decision support system. Int ] Technol Assess Health
Care. 2000;16:210-27. [PMID: 10815366]
171. Bosworth HB, Olsen MK, Dudley T, Orr M, Goldstein MK, Datta SK,
et al. Patient education and provider decision support to control blood pressure
in primary care: a cluster randomized trial. Am Heart J. 2009;157:450-6.
[PMID: 19249414]
172. Bosworth HB, Olsen MK, Goldstein MK, Orr M, Dudley T, McCant F,
et al. The veterans’ study to improve the control of hypertension (V-STITCH):
design and methodology. Contemp Clin Trials. 2005;26:155-68. [PMID:
15837438]
173. Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of
health information technology: a review of the recent literature shows predomi-
nantly positive results. Health Aff (Millwood). 2011;30:464-71. [PMID:
213833065]
174. Osheroff JA; Healthcare Information and Management Systems Society.

Improving Outcomes with Clinical Decision Support: An Implementer’s Guide.
2nd ed. Chicago: Healthcare Information and Management Systems Society;
2005.
3 July 2012 | Annals of Internal Medicine | Volume 157 ¢ Number 1|43
Annals of Internal Medicine
Current Author Addresses: Dr. Bright: 16 Kenilworth Drive, Hamp-

ton, VA 23666.
Mr. Wong: 2618 Briar Trail, Apartment 202, Schaumburg, IL 60173.

Dr. Dhurjati: D330-1 Mayo (MMC 729), 420 Delaware Street SE,
University of Minnesota, Minneapolis, MN 55455.
Ms. Bristow: 728 Irolo Street, Apartment D, Los Angeles, CA 90005.

Dr. Bastian: Health Services Research & Development, 152 Veterans
Affairs Medical Center, 508 Fulton Street, Durham, NC 27705.
Drs. Coeytaux, Hasselblad, Williams, and Sanders; Mr. Musty; Ms.
Wing; and Ms. Kendrick: Duke Evidence-based Practice Center, Duke
Clinical Research Institute, 2400 Pratt Street, Durham, NC 27705.
Dr. Samsa: Department of Biostatistics and Bioinformatics, Duke Uni-
versity School of Medicine, Durham, NC 27705.
Dr. Lobach: Klesis LLC, 6 Harvey Place, Durham, NC 27705.
Author Contributions: Conception and design: T.J. Bright, A. Wong,

J.W. Williams, G.D. Sanders, D. Lobach.
Analysis and interpretation of the data: T.J. Bright, A. Wong, R. Dhurjati,

G. Samsa, V. Hasselblad, G.D. Sanders, D. Lobach.
Drafting of the article: T.J. Bright, A. Wong, E. Bristow, G.D. Sanders,

D. Lobach.
Critical revision of the article for important intellectual content:

T.J. Bright, G. Samsa, G.D. Sanders, D. Lobach.
Final approval of the article: T.J. Bright, G. Samsa, G.D. Sanders,

D. Lobach.
Provision of study materials or patients: T.J. Bright, M.D. Musty.

Statistical expertise: G. Samsa, V. Hasselblad.
Obtaining of funding: G.D. Sanders, D. Lobach.
Administrative, technical, or logistic support: T.J. Bright, M.D. Musty,

L. Wing, A.S. Kendrick, G.D. Sanders.
Collection and assembly of data: T.J. Bright, A. Wong, R. Dhurjati,

E. Bristow, L. Bastian, R.R. Coeytaux, M.D. Musty.
Appendix Figure. Summary of evidence search and selection.
Citations identified by literature

search (n = 15 176)
MEDLINE: 12 746 Duplicates
CINAHL and PsycINFO: 1126 [—> removed
Web of Science: 1277
Manual searching: 27
Abstracts
»| excluded
(n =13 769)
Y
Articles
passed abstract
screening
(n = 1407) Articles excluded (n = 1084)
Unable to locate full text: 1
Non-English-language article: 1
Not original peer-reviewed data: 310
Poster (or other publication type providing insufficient detail): 68
No electronic CDSS intervention: 310
CDSS not implemented in clinical setting: 125
No acceptable comparator: 148
CDSS not aimed at health care providers: 19
CDSS not used to aid decision making at point of care or for a
specific care situation: 36
Not an evaluation study: 6
v Sample size <50: 24
Closed-loop system: 1
Mandatory adherence to CDSS recommendations: 16
No outcome of interest: 19
Articles passed full-text

screening and were abstracted
for KQ 1
(n = 323)
Study design other than RCT (n = 163)
Y
Articles abstracted
for KQs 24
(n = 160)
CDSS = clinical decision-support system; KQ = key question; RCT = randomized,

controlled trial.
www.annals.org 3 July 2012 | Annals of Internal Medicine | Volume 157 ¢ Number 1|W-
7

Downloaded From
: http
//annals.org/ on 10/31/2016
W-8 | 3 July 2012 |Annals of Internal Medicine

Volume 157 * Number 1
www.annals.org
Appendix Table 1. Study Characteristics*
Outcome
Clinical outcomes
Length of stay (n = 6)
Morbidity (n = 22)
Mortality (n = 7)
HRQOL (n = 6)
Adverse events (n
Health care process

measures
Recommended prevent
= 43)
care service (
:5)
ive
Recommended clinical
study (n = 29)
Recommended treati
(n = 67)
ment
Location
us: 4
Europe: 1
Multicountry: 1
us: 16
Europe: 3
Multicountry: 2
us: 6
Multicountry: 1
us: 5
Europe: 1
us: 4
Multicountry: 1
us: 29
Europe: 5
Canada: 6
Australia: 2
New Zealand:
1
us: 17
Europe: 7
Canada: 3
Australia: 1
NR: 1
uUs: 43
Europe: 18
Canada: 4
Multicountry: 1
NR: 1
General Setting
Academic: 4
Community: 1
NR: 1
Academic: 11
Community: 5
Academic and
community: 2
VA: 3
NR: 1
Academic: 4
Academic and
community: 1
VA: 1
NR: 1
Academic: 3
Community: 2
VA: 1
Academic: 4
Academic and
community: 1
Academic: 20
Community: 15
Academic and
community: 5
VA: 1
NR: 2
Academic: 10
Community: 9
Academic and
community: 5
VA: 1
NR: 4
Academic: 24
Community: 22
Academic and
community: 12
VA: 4
Academic and
VA: 1
NR: 4
Specific Setting
Inpatient: 3
Outpatient: 1
ED: 2
Inpatient: 6
Outpatient: 13
ED: 1
Inpatient and
outpatient: 1
NR: 1
Inpatient: 4
Outpatient: 2
NR: 1
Inpatient: 5
Outpatient: 1
Inpatient: 3
Outpatient: 1
Long-term
facility: 1
Inpatient: 5
Outpatient: 38
Inpatient: 2
Outpatient: 24
ED: 3
Inpatient: 13
Outpatient: 47
ED: 3
Inpatient and
outpatient: 2
Long-term
facility: 1
NR: 1
Duration
<6 mo: 1
6-12 mo: 2
2-3y:2
NR: 1
<6 mo: 1
6-12 mo: 6
1-2y:5
2-3y:6
>3y:2
NR: 2
<6 mo: 2
6-12 mo: 4
>3y:1
6-12 mo: 3
2-3y:2
NR: 1
<6 mo: 2
6-12 mo: 1
23y: 1
NR: 1
<6 mo: 4
6-12 mo: 22
1-2y:7
2-3y:3
>3y:1
NR: 6
<6 mo: 4
6-12 mo: 14
1-2y: 5
2-3y:3
NR: 3
<6 mo: 9
6-12 mo: 32
1-2y:7
2-3y: 6
>3y:2
NR: 11
Source
Local: 5
Commercial: 1
Local: 20
Commercial: 2
Local: 6
Commercial: 1
Local: 6
Local: 4
Commercial: 1
Local: 27
Commercial: 10
NR: 6
Local: 20
Commercial: 7
NR: 2
Local: 48
Commercial: 14
NR: 5
Objective
Diagnosis: 3
Pharmacotherapy: 3
Chronic disease
management: 1
Laboratory test
ordering: 2
Diagnosis: 4
Pharmacotherapy: 9
Chronic disease
management: 10
Laboratory test
ordering: 2
Other: 6
Diagnosis: 1
Pharmacotherapy: 5
Chronic disease
management: 2
Other: 2
Chronic disease
management: 5
Other: 1
Pharmacotherapy: 2
Other: 3
Diagnosis: 5
Pharmacotherapy: 7
Chronic disease
management: 11
Laboratory test
ordering: 10
Initiating discussion
with patients: 3
Other: 35
Diagnosis: 9
Pharmacotherapy: 2
Chronic disease
management: 5
Laboratory test
ordering: 16
with patients: 2
Other: 4
Diagnosis: 9
Pharmacotherapy: 45
Chronic disease
management: 25
Laboratory test
ordering: 5
Other: 9
Relation
Sync: 5
Async: 1
Sync: 16
Async: 3
Both: 1
NR: 2
Sync: 5
Async: 1
NR: 1
Sync: 4
Async: 2
Sync: 5
Sync: 40
Async: 1
NR: 2
Sync: 27
Async: 2
Sync: 58
Async:4
Both: 2
NR: 3
Response
NR: 2
Noncommittal
acknowledgment: 1
NR or assume NR: 2
NR or unclear: 1
Mandatory response: 3
Noncommittal
acknowledgment: 4
NR: 4
NR or assume NR: 11
NR: 1
NR or assume NR: 3
NR: 2
Noncommittal
acknowledgment: 2
NR or unclear: 2
No response: 1
NR or assume NR: 3
Justify: 3
NR: 11
Noncommittal
acknowledgment: 5
Mandatory and
justify: 1
NR or assume NR: 19
Justify: 2
NR: 5
Noncommittal
acknowledgment: 1
NR or assume NR: 15
Justify: 5
NR: 43
Noncommittal
acknowledgment: 6
Format
Integrated: 3
Standalone: 1
Paper: 2
Integrated: 8
Fax or computer
printout: 7
Standalone: 6
Integrated and
printout: 1
Integrated: 4
Standalone: 2
Integrated and
pager: 1
Integrated: 2
Fax or computer
printout: 3
Integrated and
printout: 1
Integrated: 2
Integrated and
pager: 1
Standalone: 1
NR: 1
Integrated: 13
Fax or computer
printout: 20
Standalone: 5
Online: 2
Integrated and
printout: 2
Online and
printout: 1
Integrated: 20
Fax or computer
printout: 3
Standalone: 3
Other: 3
Integrated: 40
Fax or computer
printout: 9
Standalone: 11
Online: 1
Combination: 3
NR: 3
Mode
System: 6
System: 17
User: 2
NR: 3
System: 16
NR: 1
-—
System:
System: 3
User: 1
NR: 1
System: 35
User: 6
NR: 2
System: 21
User: 5
NR: 3
System: 54
User: 9
Both: 1
NR: 3
Quality
Good: 6
Good: 13
Fair: 7
Poor: 2
Good: 6
Fair: 1
Good: 3
Fair: 2
Poor: 1
Good: 3
Fair: 1
Poor: 1
Good: 20
Fair: 16
Poor: 7
Good: 16
Fair: 9
Poor: 4
Good: 35
Fair: 24
Poor: 8
Downloaded From
www.annals.org
http
3 July 2012 |Annals of Internal Medicine
Outcome
User workload and effici
outcomes
Effect on user knowledge

(n =5)
Number of patients seen

per unit time (n =
Clinician workload (n
Efficiency (n = 7)
Relationship-centered
outcomes
Patient satisfaction (n = 6)
Economic outcomes
Cost (n = 22)
Cost-effectiveness (n
Use and implementation
outcomes
Health care provider

acceptance (n =
Location
us: 1
Europe: 2
Canada: 1
Multicountry: 1
NA
NA
us: 5
Canada: 1
Multicountry: 1
us: 4
Canada: 1
NR: 1
us: 14
Europe: 6
Multicountry: 1
NR: 1
Europe: 2
Canada: 4
us: 17
Europe: 5
Canada: 2
General Setting
Community: 3
NR: 2
NA
NA
Academic: 4
Community: 2
NR: 1
Academic: 3
Community: 2
NR: 1
Academic: 11
Community: 10
NR: 1
Academic: 4
Community: 2
Academic: 10
Community: 5
Academic and
community: 3
VA: 2
Academic and
VA: 1
NR: 3
Specific
Setting
Outpatient: 4
NR: 1
NA
NA
Inpatient: 3
Outpatient: 3
NR: 1
Inpatient: 1
Outpatient: 4
ED: 1
Inpatient: 5
Outpatient: 17
Outpatient: 6
Inpatient: 3
Outpatient: 19
ED: 1
Long-term
facility: 1
Duration
<6 mo: 1
6-12 mo: 3
NR: 1
NA
NA
<6 mo: 4
6-12 mo: 1
12 y:1
2-3y:1
<6 mo: 1
2-3y:3
NR: 2
<6 mo: 5
6-12 mo: 11
12 y:1
2-3y:3
NR: 2
<6 mo: 1
6-12 mo: 4
12y: 1
<6 mo: 3
6-12 mo: 12
12y:5
2-3y:1
NR: 3
Source
Local: 2
Commercial: 2
NR: 1
NA
NA
Local: 4
Commercial: 3
Local: 4
Commercial: 1
NR: 1
Local: 17
Commercial: 5
Local: 3
Commercial: 2
NR: 1
Local: 21
Commercial: 2
NR: 1
Objective
Diagnosis: 1
Chronic disease
management: 1
Laboratory test
ordering: 1
with patients: 1
Other: 3
NA
NA
Pharmacotherapy: 2
Chronic disease
management: 1
Laboratory test
ordering: 2
Other: 3
Diagnosis: 2
Chronic disease
management: 2
Laboratory test
ordering: 1
with patients: 1
Other: 2
Diagnosis: 5
Pharmacotherapy: 5
Chronic disease
management: 9
Laboratory test
ordering: 6
with patients: 2
Other: 7
Diagnosis: 1
Pharmacotherapy: 1
Chronic disease
management: 1
Other: 4
Diagnosis: 3
Pharmacotherapy: 9
Chronic disease
management: 5
Laboratory test
ordering: 4
with patients: 1
Other: 12
Relation
Sync: 4
NR: 1
NA
NA
Sync: 5
Async:1
Both: 1
Sync: 6
Sync: 20
Async: 2
Sync: 6
Sync: 21
Async: 2
Both: 1
Response
NR: 1
NR or assume NR: 3
NA
NA
NR: 1
NR or assume NR: 3
NR: 3
NR or assume NR: 3
Justify: 2
NR: 5
Noncommittal
acknowledgment: 3
NR or assume NR: 7
NR: 4
NR or assume NR: 1
Justify: 7
NR: 3
Noncommittal
acknowledgment: 1
Mandatory and justify: 1
NR or assume NR: 8
Format
Integrated: 1
Standalone: 1
Online: 2
NR: 1
NA
NA
Integrated: 3
Standalone: 1
Online: 1
Pager: 1
Online and e-mail:

1
Integrated: 3
Standalone: 1
Online: 1
Fax or computer
printout: 1
Integrated: 11
Standalone: 4
Fax or computer
printout: 3
Integrated and fax

or computer
printout: 1
Other: 3
Integrated: 1
Standalone: 1
Fax or computer
printout: 4
Integrated: 11
Standalone: 3
Fax or computer
printout: 6
Other: 4
Mode
User: 3
NR: 2
NA
NA
System: 4
User: 2
NR: 1
System: 4
NR: 2
System: 17
User: 4
NR: 1
System: 5
User: 1
System: 18
User: 6
Quality
Fair: 4
Poor: 1
NA
NA
Good: 3
Fair: 4
Good: 4
Fair: 1
Poor: 1
Good: 10
Fair: 7
Poor: 5
Good: 1
Fair: 5
Good: 9
Fair: 11
Poor: 4
Volume 157 * Number 1 |W-9
Downloaded From
: http
W-10 | 3 July 2012 |Annals of Internal Medicine
Volume 157 * Number 1
www.annals.org
Outcome Location General Setting Specific Duration Source Objective Relation

Response Format Mode Quality
Setting
Healt
h care provider use (n us: 9 Academic: 2 Inpatient: 1 6-12 mo: 13 Local: 13
Diagnosis: 3 Sync: 17 Mandatory response: 2 Integrated: 13 System: 7 Good: 5
17)
Europe:7 Community: 9 Outpatient: 16 1-2y: 4 Commercial: 3 Pharmacotherapy: 9

Justify: 1 Standalone: 2 User: 8 Fair: 10
Canada: 1 Academic and NR: 1 Chronic disease NR: 4 Fax or computer Both: 1 Poor: 2
community: 3 management: 6 NR or assume NR: 10 printout: 1 NR: 1
VA: 1 Laboratory test Other: 1
NR: 2 ordering: 4
Other: 4
Async = asynchronous; ED = emergency department; HRQOL = health-related quality of

life; justify = justification for not adhering to the recommendation; NA = not
applicable; NR = not reported; sync = synchronous;
US = United States; VA = Veterans Affairs.
* Numbers signify the number of studies.
Appendix Table 2. Examples of Clinical Decision-Support Interventions
Outcome
Clinical outcomes
Length of stay
Morbidity
Mortality
HRQOL
Adverse events
Health care process measures

Recommended preventive care
service ordered or completed
Recommended clinical study
ordered or completed
Recommended treatment
ordered or prescribed
User workload and efficiency

outcomes
Effect on user knowledge
Number of patients seen per unit

time
Clinician workload
Efficiency
Relationship-centered outcomes
Patient satisfaction
Economic outcomes
Cost
Cost-effectiveness
Use and implementation outcomes

Health care provider acceptance
Health care provider satisfaction
Health care provider use
Implementation
Studies Included, n;
Meta-analysis Result
(95% ClI)
5; RR, 0.96 (0.88-1.05)
16; RR, 0.88 (0.80-0.96)
6; OR, 0.79 (0.54-1.15)

NA
5; RR, 1.01 (0.90-1.14)
25; OR, 1.42 (1.27-1.58)

20; OR, 1.72 (1.47-2.00)
46; OR, 1.57 (1.35-1.82)
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
Intervention Example
Standalone system to decrease inappropriate antimicrobial use by recommending the 3
“best”
antibiotic regimens reported that the intervention group had a significantly lower
length of
stay than the control group (RR, 0.9082 [95% CI, 0.8392-0.9828]) (21)
Guideline-based diabetes recommendations to improve cholesterol, creatinine,

urinary protein,
and hemoglobin A;. measurement and found that the overall number of
hospitalizations
was significantly lower in the intervention group for all participants (0.17 vs.
0.20; P =
0.01) and the number of ED visits was significantly lower for all intervention
participants
(0.27 vs. 0.36; P < 0.001) (31, 32)
Treatment reminders to improve the appropriate use of B-blockers for patients with
CHF
found a reduction in mortality by 12% (P = 0.050) (RR, 0.12 [CI, 0.016-0.87]) (22)
Evidence-based treatment recommendations for the management of chronic heart

failure
found significant improvements in the mental component score for intervention
patients
compared with patients in the control group at 6 and 12 mo (38)
Alerts to detect potentially inappropriate antimicrobial therapy used the frequency

of
Clostridium difficile testing as an indicator for the presence of diarrhea, and
adverse effects
of antimicrobial use reported that fewer intervention patients experienced diarrhea
as an
adverse effect of antimicrobial therapy (5.7% vs. 6.6%; P = 0.21) (40)
Alerts to identify patients at risk for DVT reported more prophylactic measures in
the
intervention group than in the control group (33.5% vs. 14.5%; P < 0.001) (17)
Guideline-base§ reminders to discuss chlamydia testing for women aged 16 to 24 y to

improve the rate of chlamydia testing significantly increased across intervention
and control
groups, but the intervention clinics had a greater increase in testing (108)
Prescribing alerts that targeted potentially inappropriately prescribed medications

for elderly
patients reported that there were significantly fewer inappropriate prescriptions
in the
intervention group than in the control group (OR, 0.59 [CI, 0.41-0.85]; P = 0.006)
(147)
Providers reported that in 62% of sessions, the use of an information retrieval

tool embedded
in an EHR system that provided access to topic or nonspecific links to clinical
resources to
aid in answering clinicians' questions at the point of care enhanced their
decisions or
knowledge (156)
NA
NA
Clinicians who received CDSS alerts spent roughly 1 h less each day resolving
inappropriate
antibiotic prescriptions in the intervention group than in the control group of the
trial (40)
Patients who were treated by intervention providers who used a discharge planning
application had a higher perception of discharge preparedness and satisfaction with
medication information (23, 24)
Guideline-based diabetes recommendations to improve appropriate testing reported a

significant reduction in hospitalization expenses for all participants in the
intervention group
($3113.19 vs. $3480.14; P = 0.02), and a significant reduction in ED expenses was
also
found for all participants in the intervention group ($414.30 vs. $301.51; P <
0.001)
(31, 32)
Reminders with performance of risk estimation and choice of drugs; reminders

triggered by
elevated blood pressure or low-density lipoprotein cholesterol levels in patients
that
provided prescribing recommendations for antihypertensive and cholesterol-lowering
drug
therapy estimated that the cost of using the CDSS was $183 per additional patient
initiating thiazide therapy (53, 54)
Prescribing alerts for heavily marketed hypnotic medications that included

alternative
treatment suggestions and information on prescribing, patient education materials,
and cost
reported that only 23% of providers believed that the recommendations changed their
prescribing decisions (119)
Clinical practice reminders to assess HIV risk or offer testing reported that 61%
of providers
described the reminders to be “useful” in a postintervention survey (105)
Prescribing reminders for antihypertensive medications found that, during the 2-y
evaluation
period in which the CDSS intervention was displayed, providers interacted with the
intervention in 57% of the visits (n = 528 of 929) (171, 172)
Predicted probabilities of test abnormalities presented to providers when ordering

diagnostic
tests significantly reduced costs, and upon discontinuation of the intervention,
the
postintervention levels of ordering returned to preintervention levels (160)
CDSS = clinical decision-support system; CHF = congestive heart failure; DVT = deep
venous thrombosis; ED = emergency department; EHR = electronic health record;
HRQOL = health-related quality of life; NA = not applicable; OR = odds ratio; RR =
relative risk.
www.annals.org
3 July 2012 | Annals of Internal Medicine | Volume 157 ¢ Number 1 |W-11

Bright Et Al. - 2012 - Effect of Clinical Decision-Support Systems A Sys PDF

Uploaded by

Copyright:

Available Formats

Bright Et Al. - 2012 - Effect of Clinical Decision-Support Systems A Sys PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bright Et Al. - 2012 - Effect of Clinical Decision-Support Systems A Sys PDF

Uploaded by

Copyright:

Available Formats

Annals of Internal Medicine

Effect of Clinical Decision-Support Systems

Gillian D. Sanders, PhD; and David Lobach, MD, PhD

Background: Despite increasing emphasis on the role of clinical

Purpose: To evaluate the effect of CDSSs on clinical outcomes,

Data Sources: MEDLINE, CINAHL, PsycINFO, and Web of Science

Study Selection: Investigators independently screened reports to

Data Extraction: Investigators extracted data about study design,

Data Synthesis: 148 randomized, controlled trials were included.

Both commercially and locally developed CDSSs improved health

Limitations: Studies were heterogeneous in interventions, popula-

Conclusion: Both commercially and locally developed CDSSs are

Ann Intern Med. 2012;157:29-43.

Despite increasing emphasis on clinical decision-

Classic CDSSs include alerts, reminders, order sets,

Until recently, most studies of CDSSs came from 4

Downloaded From: http://annals.org/ on 10/31/2016

Institute) (2). Although several reviews have examined the

We developed and followed a standard protocol for

We searched for studies done between January 1976

We identified randomized trials of CDSSs imple-

© 2012 American College of Physicians [ 29

REVIEW | Effect of Clinical Decision-Support Systems

verse events), health care process (recommended preventive

Data Extraction and Quality Assessment

Data related to study setting and design, sample char-

Data Synthesis and Analysis

A priori—defined outcomes believed to be important

Role of the Funding Source

Primary funding was provided by the Agency for

We screened 15 176 abstracts, evaluated 1407 full-text

30 | 3 July 2012 |Annals of Internal Medicine | Volume 157 ® Number 1

Downloaded From: http://annals.org/ on 10/31/2016

22 (15%) measured costs. Many studies (7 = 51) were

The Table summarizes the most important findings

Several studies assessed morbidity outcomes (14-38),

Many studies evaluated locally developed interventions

Comparators included usual care or no CDSS and the

Meta-analysis of these heterogeneous studies (7 = 10)

Effect of Clinical Decision-Support Systems REVIEW

Table. Summary of Evidence, by Outcome

Outcome Evidence Studies (Quality Meta-analysis Studies

Mortality Low 7 (6 good, 1 fair) OR, 0.79 6

HRQOL Low 6 (3 good, 2 fair, NA -

Adverse events Low 5 (3 good, 1 fair, RR, 1.01 5

Health care process measures

Recommended preventive High 43 (20 good, OR, 1.42 25

Recommended clinical study Moderate 29 (16 good, 9 fair, OR, 1.72 20

Recommended treatment High 67 (35 good, OR, 1.57 46

User workload and efficiency outcomes

Number of patients seen Insufficient 0 NA -

Clinician workload Insufficient 0 NA =

Efficiency Low 7 (3 good, 4 fair) NA -

Other Substantial Findings

Limited evidence that CDSSs that automatically delivered

Modest evidence from academic and community

Limited evidence that CDSSs integrated in CPOE or EHR

Limited evidence from ambulatory settings that

Limited evidence from academic settings that CDSSs that