(JURNAL, Eng) Reducing Potentially Inappropriate Prescriptions For Older Patients Using Computerized Decision Support Tools, Systematic Review

JOURNAL OF MEDICAL INTERNET RESEARCH Monteiro et al
Review
Reducing Potentially Inappropriate Prescriptions for Older Patients

Using Computerized Decision Support Tools: Systematic Review
Luís Monteiro1,2, MD; Tiago Maricoto3,4, MD; Isabel Solha5, MD; Inês Ribeiro-Vaz2,6,7, PhD; Carlos Martins2,7, PhD;
Matilde Monteiro-Soares2,7, PhD
1
Esgueira+ Family Health Unit, Aveiro Healthcare Centre, Aveiro, Portugal
2
Center for Health Technology and Services Research, Faculty of Medicine, University of Porto, Porto, Portugal
3
Aveiro-Aradas Family Health Unit, Aveiro Healthcare Centre, Aveiro, Portugal
4
Faculty of Health Sciences, University of Beira Interior, Covilhã, Portugal
5
Terras de Souza Family Health Unit, Paredes, Portugal
6
Porto Pharmacovigilance Centre, Faculty of Medicine, University of Porto, Porto, Portugal
7
Department of Community Medicine, Information and Decision in Health, Faculty of Medicine, University of Porto, Porto, Portugal
Corresponding Author:
Luís Monteiro, MD
Esgueira+ Family Health Unit, Aveiro Healthcare Centre
Rua Pedro Vaz de Eça
Aveiro
Portugal
Phone: 351 00351234312890
Email: [email protected]
Abstract
Background: Older adults are more vulnerable to polypharmacy and prescriptions of potentially inappropriate medications.
There are several ways to address polypharmacy to prevent its occurrence. We focused on computerized decision support tools.
Objective: The available literature was reviewed to understand whether computerized decision support tools reduce potentially
inappropriate prescriptions or potentially inappropriate medications in older adult patients and affect health outcomes.
Methods: Our systematic review was conducted by searching the literature in the MEDLINE, CENTRAL, EMBASE, and Web
of Science databases for interventional studies published through February 2018 to assess the impact of computerized decision
support tools on potentially inappropriate medications and potentially inappropriate prescriptions in people aged 65 years and
older.
Results: A total of 3756 articles were identified, and 16 were included. More than half (n=10) of the studies were randomized
controlled trials, one was a crossover study, and five were pre-post intervention studies. A total of 266,562 participants were
included; of those, 233,144 participants were included and assessed in randomized controlled trials. Intervention designs had
several different features. Computerized decision support tools consistently reduced the number of potentially inappropriate
prescriptions started and mean number of potentially inappropriate prescriptions per patient. Computerized decision support tools
also increased potentially inappropriate prescriptions discontinuation and drug appropriateness. However, in several studies,
statistical significance was not achieved. A meta-analysis was not possible due to the significant heterogeneity among the systems
used and the definitions of outcomes.
Conclusions: Computerized decision support tools may reduce potentially inappropriate prescriptions and potentially inappropriate
medications. More randomized controlled trials assessing the impact of computerized decision support tools that could be used
both in primary and secondary health care are needed to evaluate the use of medication targets defined by the Beers or STOPP
(Screening Tool of Older People’s Prescriptions) criteria, adverse drug reactions, quality of life measurements, patient satisfaction,
and professional satisfaction with a reasonable follow-up, which could clarify the clinical usefulness of these tools.
Trial Registration: International Prospective Register of Systematic Reviews (PROSPERO) CRD42017067021;
https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42017067021
(J Med Internet Res 2019;21(11):e15385) doi: 10.2196/15385
https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 1

(page number not for citation purposes)
XSL• FO
RenderX
KEYWORDS
deprescriptions; medical informatics applications; potentially inappropriate prescription; potentially inappropriate medication;
computerized decision support
have been published since then [27]. This systematic review

Introduction aims to clarify whether CDS tools can help in reducing PIPs or
The older adult population is increasing in developed countries PIMs to improve clinical outcomes in older adults.
[1], and people worldwide are living longer [2,3]. According
to the World Health Organization, people aged 60 years and Methods
older in 2020 will outnumber children younger than 5 years. In
2050, the world’s population aged 60 years and older is expected
Eligibility Criteria
to total 2 billion [2]. The systematic review was conducted according to a protocol
previously published [28] and registered in PROSPERO
The aging of populations increases the pressure on health care (International Prospective Register of Systematic Reviews;
systems, which should be aligned with the needs of older CRD42017067021). We searched for interventional controlled
populations [4]. Older patients are more likely to have more studies (type of study) with participants aged 65 years or older
than one chronic condition, known as multimorbidity [5,6]. The (population) that assessed whether CDS tools (intervention)
prevalence of multimorbidity is more than 90% in older patients could diminish PIM (outcome). Moribund or terminal
[5]. Having more than one chronic condition requires the use participants were excluded along with those requiring palliative
of several medications. Thus, older adults are more vulnerable care. No other restriction was applied.
to polypharmacy [7], meaning the use of multiple drugs
administered to the same patient [8,9], in addition to Search Methods
prescriptions of potentially inappropriate medications (PIMs) We searched MEDLINE, CENTRAL, EMBASE, and Web of
[10-12]. A PIM can be described as a medication use that has Science for studies published through February 2018 without
potentially more risks than benefits with a safer alternative language restrictions. Specific queries were used according to
available [10]. each database’s requirements that were described in detail
Potentially inappropriate prescription (PIP) is a broader concept elsewhere [29]. Trial registries, different types of grey literature,
than PIM, because it includes over-, under-, and misprescribing and contact with specialists in the field were also performed.
(eg, inappropriate dose or duration). It is defined as “the The reference lists of all included studies were searched to
prescribing of medication that could introduce a significant risk identify any potentially pertinent study that might not have been
of an adverse event, in particular when there is an equally or identified by previous methods. References were checked from
more effective alternative with lower risk available” [13]. previously published systematic reviews.
Due to changes in pharmacokinetics and pharmacodynamics, Selection Process

older people are more prone to drug interactions and adverse Articles were selected by applying the criteria to the title and
drug reactions [14,15]. Adverse drug reactions are considered abstract of each study. Studies that were selected at this stage
a public health problem in older patients and a cause of disability were then assessed in their entirety. Each stage was conducted
and mortality [15]. Deprescribing is defined as “the process of by two researchers blindly and independently. Two reviewers
withdrawal of inappropriate medication, supervised by a health (LM and TM) examined the titles and abstracts and did the
care professional, with the goal of managing polypharmacy and full-text screening. When disagreement occurred, it was resolved
improving outcomes” [16]. through consensus.
There are several ways to address polypharmacy to prevent its Data Collection Process
occurrence [17-23]. This review focused on computerized For all the included studies, characterization of data and results
decision support (CDS) tools. Bates et al [24] defined CDS were exported into a datasheet by one of the authors (LM) and
systems as computer-based systems providing “passive and confirmed by the other (MS).
active referential information as well as reminders, alerts, and
guidelines.” Payne [25] added that CDS tools can be defined Type of Data Collected
as “computer applications designed to aid clinicians in making Studies were characterized according to setting, intervention,
diagnostic and therapeutic decisions in patient care.” CDS tools comparison definition, study duration, number of included
may have a positive impact on health care, such as reducing participants overall and in each study group, the proportion of
physicians’ orders of unnecessary tests [26]. missing data, participants’ mean age, the proportion of male
Previous studies reviewed such strategies, such as individuals, and deprescribing target. Outcomes retrieved from
multidisciplinary team medication reviews, pharmacist each study were categorized as PIP- or PIM-related and by
medication reviews, computerized clinical decision support overall number of prescriptions, adverse drug reactions, and
systems, and multifaceted approaches and reported substantial potential drug-drug interactions.
heterogeneity in the included studies, but did not focus on CDS Analysis of Results and Assessment of the Risk of Bias
[19,21]. One systematic review that did focus on CDS systems
included studies published only through 2012, and new studies Possible bias in randomized controlled trials (RCTs) was
independently identified using the Cochrane Collaboration Risk
XSL• FO
RenderX
of Bias tool [29] by two researchers (TM and LM). This The characteristics of the included studies are described in Table
assessment was confirmed by other authors (IV and MS). Risk 1. More than half (10/16) of the included studies were RCTs,
of bias was determined with regard to random sequence one was a crossover study, and five were pre-post intervention
generation, allocation concealment, blinding of participants and studies. Most studies were conducted in North America (Canada
personnel, blinding of outcome assessments, incomplete and United States; n=11) [30-40]. The remaining were conducted
outcome data, selective outcome reporting, and other biases. in Europe (n=5) [41-45].
The included articles did not permit the performance of a Six studies were conducted exclusively in secondary health care
meta-analysis because there were not a minimum of three studies institutions [35,37,38,40,44,45]. In two studies, only emergency
using the same deprescribing target. Thus, only a narrative department participants were included [33,39]. In total, six
synthesis was performed. We have summarized the main studies were performed exclusively in primary health care
features and results of all the included studies, discussed their institutions [30-32,41-43], one study took place in a health
limitations, and proposed future research avenues. maintenance organization [34], and one study included
participants from both secondary and primary health care
Results institutions [36]. Six studies took place at teaching hospitals
[36-38,40,44,45].
Description of the Studies
Most commonly, the standard of care was the only comparator
Using our search strategy, 3756 articles were identified through (n=11). The interventional design was always based on a CDS
MEDLINE, Central, EMBASE, and Web of Science databases. tool, which was usually included in the electronic medical record
One article was identified through contact with specialists. After with several different features. In some cases (n=6), complex
duplicates were removed, 2819 articles remained. The titles and interventions were performed that included training and
abstracts were screened, and 2767 studies were excluded. Of engagement sessions and/or leaflet provision.
these, 52 articles were selected to assess eligibility and their
full text was analyzed. Of these, 36 articles were excluded. The RCTs had an inclusion period ranging from 3 to 30 months
Ultimately, we included 16 studies in our systematic review. (see Table 2). The crossover study included four on-off periods
No new article was found by searching in the included studies’ with a 6-week duration [33]. The pre-post intervention studies
reference lists, trial registries, or grey literature. The article frequently compared different time periods.
selection process and reasons for exclusion are described in
Figure 1.

XSL• FO
RenderX
Figure 1. Flow diagram on search and article inclusion, according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses
(PRISMA) Statement.

XSL• FO
RenderX
Table 1. Descriptions of the included studies in the systematic review (N=16).

Author, year; (study); country Setting Comparator Intervention Deprescribing target
Randomized controlled trials
Tamblyn et al [30], 2003; PHCa Usual careb Computerized decision support tool PIPc (159 clinically relevant PIPs in
Canada providing alert identified problem the elderly defined by expert consen-
+ presented possible consequences sus)
+ provided alternative therapy
Price et al [31], 2017; PHC (8 GPd) Usual care Clinical decision support tool PIPs (40 STOPP criteria)
Canada showing alert with specific STOPPe
guideline content in electronic
medical record
Avery et al [41], 2012; PHC (72 GP) Computer-gener- PINCER; comparator + pharmacist- PIPs on NSAIDsf, beta blockers,
(PINCER); UK ated simple feed- led information technology complex
ACEg inhibitors, or loop diuretics
back intervention
Erler et al [42], 2012; Ger- PHC (46 GP) Usual care Interactive 1-hour workshop for Prescription exceeding recommended
many physicians on detection and manage- standard; daily dosage >30% or rec-
ment of CKDh + provision of desk- ommended; maximum daily dose in
top checklist of medications to be CKD patients
reduced or avoided + patient infor-
mation leaflets + training in the use
of software “DOSING”
Clyne et al [43], 2015; PHC (21 GP) Usual care + sim- Comparator + academic detailing PIPs using 28 criteria from the study
(OPTI-SCRIPT); Ireland ple, patient-level with pharmacist + medicine review
PIP postal feed- with Web-based pharmaceutical
back treatment algorithms + leaflets
Cossette et al [40], 2017; SHCi (teaching Usual care KTj strategy; distribution of educa- 7 PIMsk based Beers and STOPP
Canada hospital) tional materials + in-services by geriatric criteria and drugs with anti-
geriatricians + computerized alert cholinergic properties or acting on the
systems pharmacist-physician central nervous system
Fried et al [32], 2017; PHC (Veterans Af- Usual care only 2 Web apps: (1) extracts information Medication appropriateness based on
(TRIM); USA fairs; medical cen- and usual care on medications and chronic condi- range of criteria, including feasibility
ter) with telephonic tions from the electronic health in context of patient’s cognition and
patient assess- record, (2) interface for data chart social support, potential overtreatment
ment review and telephonic patient assess- of DMl or hypertension, “traditional”
ment + a set of automated algo- PIMs according to Beers and STOPP
rithms evaluating medication appro- criteria, inappropriate renal dosing,
priateness + patient-specific medica- and patient report of adverse medica-
tion management feedback report tion effects
for the clinician
O’Sullivan et al [44], SHC (teaching Usual medical Clinical decision support software Medicines associated with “nontriv-
2016; Ireland hospital) and pharmaceuti- supported structured pharmacist re- ial” adverse drug reactions (according
cal care view of medication designed to op- to WHO)
timize geriatric pharmaceutical care
Terrel et al [33], 2009; EDm (teaching Computerized; Computer-assisted decision support 9 high-use and high-impact PIMsn
USA hospital) physician order alert when PIM was being pre-
entry without scribed + rationale + recommended
alerts safer substitute therapies. If physi-
cian chose to continue, second menu
displayed to query most important
reason
Raebel et al [34], 2007; HMOo (18 medical Usual care Medication alert generated from Newly prescribed PIMs based on the
USA offices + 21 phar- PIMS not allowing prescription la- Beers, Zhan and Kaiser Performance
macies) bel to be printed until the pharmacist Care Management Institute lists of
actively determined whether pre- medications to be avoided in older
scription should be dispensed; peoplep
pharmacists should communicate
notifications to prescribing clini-
cians
Crossover studies

XSL• FO
RenderX
Author, year; (study); country Setting Comparator Intervention Deprescribing target

Peterson et al [35], 2005; SHC Usual computer- Guided dosing of psychotropic Benzodiazepines, opiates, and neu-
USA ized order entry medication integrated in Brigham roleptics
Integrated Computer System
Pre-post intervention studies
Ruhland et al [36], 2017; SHC + PHC; (1 Usual care Clinical decision support system PIMs on glyburide
USA teaching hospital + creating an alert + rational and; alter-
2 community hospi- native medication through Epic (an
tal + 31 clinics) integrated electronic medical record)
Mattinson et al [37], 2010; SHC (teaching Usual care Medication-specific warning system PIMs on medications not recommend-
USA hospital) (advised alternative medication or ed for use in older patients (not recom-
dose reduction) mended medications) and those for
which only a reduced dose was ad-
vised (dose-reduction medications)
Lester et al [38], 2015; SHC (teaching Computerized Computerized; physician order entry PIPs on diphenhydramine, metoclo-
USA hospital) physician order with pop-up alerts for selected PIPs pramide, and antipsychotics
entry without containing links to articles relevant
alerts to the alert
Ghibelli et al [45], 2013; SHC (teaching Analysis without Computer-based application (IN- PIMs from 2003 Beers Criteria; poten-
(INTERcheck); Italy hospital) any interference TERCheck) that collects, stores and tial DDIsq; and Anticholinergic Cog-
automatically; provides drug infor- nitive Burden Scale
mation to reduce or prevent PIPs
Stevens et al [39], 2017; ED (10 Veterans Usual care EQUiPPED interventions: education PIMs from 2012 Beers Criteria cate-
(EQUiPPED); USA Affairs; medical + informatics-based clinical decision gory 1 (to avoid in all older adults)
centers) support + individual provider feed-
back
a
PHC: primary health care.
b
Each physician was given a computer, printer, health record software, and access to the internet.
c
PIP: potentially inappropriate prescription.
d
GP: general practice.
e
STOPP: Screening Tool of Older People’s Prescriptions.
f
NSAID: nonsteroidal anti-inflammatory drug.
g
ACE: angiotensin-converting enzyme.
h
CKD: chronic kidney disease.
i
SHC: secondary health care.
j
KT: knowledge translation.
k
PIM: potentially inappropriate medication.
l
DM: diabetes mellitus.
m
ED: emergency department.
n
High-use and high-impact PIMs: promethazine, diphenhydramine, diazepam, propoxyphene with acetaminophen, hydroxyzine, amitriptyline,
cyclobenzaprine, clonidine, indomethacin.
o
HMO: health maintenance organization.
p
Examples of medications to be avoided in older people: amitriptyline, chlordiazepoxide, chlorpropamide, diazepam, doxepin, flurazepam, aspirin in
combination with hydrocodone or oxycodone, ketorolac, oral meperidine, and piroxicam.
q
DDI: drug-drug interaction.

XSL• FO
RenderX
Table 2. Characterization of the included studies in the systematic review, including study type, study duration, sample size, and participant demographics
(N=16).
Study Study duration Sample Participants, n Outcome missing
(months); date range size, N data, n (%)
Age (years), mean (SD) Gender (male), n (%)
Tamblyn et al 13; (01/1997-02/1998) 12,560 Ca: 6276; Ib: C: 75 (6); I: 75 (6) C: 2248 (36); I: 2439 N/Rc
[30] 6284 (39)
Price et al [31] 8; (02-10/2015) 81,905 C:37,615; I: N/R; all >65 years N/R N/R
44,290
Avery et al [41] 6 (and 12) 480,942 C: 37,659; I: N/R N/R C: 22 (0.06); I: 28
34,413 (0.08) for outcome
3
Erler et al [42] 6 404 C: 206; I: 198 C: 80 (9); I: 81 (6) C: 63 (31); I: 81 (41) C: 9 (4); I: 0 (0)
Clyne et al [43] 6; (10/2012-09/2013) 196 C: 97; I: 99 C: 76 (5); I: 77 (5) C: 50 (52); I: 55 (56) C: 3 (3); I: 3 (3)
Cossette et al 10 weeks; (09/2015- 321 C: 133; I: 139 C: 81 (7); I: 82 (8) C: 53 (41); I:48 (38) C: 5 (4); I: 13 (9)
[40] 12/2015)
Fried et al [32] 3; (10/2014-01/2016) 156 C1: 36; C2: <70 years C: 25 (39); I: C: 63 (99); I: 63 (99) C1: 4 (11); C2:7
39; I: 81 27 (42) (18); I: 17 (21)
O’Sullivan et al 13; (06/2011-07/2012) 737 C: 361; I: 376 C: 78b; (IQR 72-84); I: C: 190 (51); I: 180 (50) C: 17 (5); I: 17 (5)
[44] 77; (IQR 71-83)
Terrel et al [33] 30; (12/01/2005- 5162 C: 2515; I: C: 74 (7); I: 74 (7) C: 880 (35); I: 929 (35) N/R
07/07/2007) 2647
Raebel et al 12; (18/05/2005- 59,680 C: 29,840; I: C: 74; (5-95 percentile C: 12,843 (43); I: N/R
[34] 17/05/2006) 29,840 66-88); I: (5-95 per- 12704 (43)
centile 66-88)
Crossover studies
Peterson et al 4 × 6 week on-off pe- 3718 C: 1925; I: C: 75 (7); I: 75 (7) C: 905 (47); I: 843 (47) N/R
[35] riods; (08/10/2001- 1793
16/05/2002)
Ruhland et al 3 + 3; (Bd: N/R 101 patients 75 N/R N/Af
[36] 01/12/2014- with activated
alert
28/02/2015); Ae:
01/03/2015-
31/05/2015)
Mattison et al 6 + 41.5; (B: 1/06- N/R N/R N/R; all >65 years N/R N/R
[37] 29/11/2014; A:
17/03/2015-
30/08/2008)
Lester et al [38] 12 + 24; (B: Q2 2010; 29,465 B: 6604; A: <75 years; B: 5279 (80); N/R N/R
A: Q2s 2011-2013) 22,861 A: 15,633 (68)
Ghibelli et al 2 + 2; (B: 04 to 134 B: 74; A: 60 B: 81; A: 81 B: 27 (36); A: 25 (42) B: 0 (0); A: 0 (0)
[45] 05/2012; A: 06 to
07/2012)
Stevens et al >6 + >12 N/R N/R N/R; all >65 years N/R N/R
[39]
a
C: comparator group.
b
I: intervention group.
c
N/R: not reported.
d
B: before.
e
A: after.
f
N/A: not applicable.

XSL• FO
RenderX
A total of 233,144 participants were included and assessed in specifically the target, such as benzodiazepines, opiates, and
RCTs (mean sample size: 21,199; range 196-72,072 neuroleptics [35]; glyburide [36]; nonsteroidal anti-inflammatory
participants). The crossover study included 3718 individuals. drugs (NSAIDs), beta blockers, angiotensin-converting enzyme
The pre-post intervention studies included more than 29,700 (ACE) inhibitors, or loop diuretics [41]; and diphenhydramine,
participants. However, some studies did not report a raw number metoclopramide, and antipsychotics [38].
of participants included in each study period. There was no
information regarding whether missing data influenced the
Results of the Studies
outcome assessment in eight studies (50%). The main results of the included studies are described in Tables
3 and 4. Several definitions and units were used to measure the
According to our inclusion criteria, all individuals were older impact of CDS tools on changes in PIP and PIM drugs (overall
than 65 years of age. The mean age in the selected studies was or concerning specific drugs). Studies assessed the following
approximately 75 years. Females were often more prevalent, PIP- or PIM-related outcomes: number of PIMs started per 1000
especially in larger studies. visits [30], number of PIMs discontinued per 1000 visits [30],
The deprescribing target varied among the studies, and several proportion of discontinued PIMs [30], percentage of PIMs [43],
papers used more than one criterion [30,32-34,40,45]. PIM was mean number of PIMs, risk of receiving a prescription for a
defined in some papers using internationally recognized criteria, drug exceeding the recommended maximum dose [42], risk of
such as the Beers Criteria (n=5) [32,34,39,40,45], the Screening receiving a prescription for a drug exceeding the recommended
Tool of Older People’s Prescriptions (STOPP) criteria (n=3) standard doses [42], proportion of reconciliation errors corrected
[31,32,40], and the Anticholinergic Cognitive Burden Scale [32], proportion of recommendations implemented [32,33],
(n=1) [45]. In other studies (n=4), some group medications were proportion of patients with at least one PIM, and/or proportion
of all prescribed medications that were PIM [33].

XSL• FO
RenderX
Table 3. Results of the included studies including changes in potentially inappropriate prescriptions or medications (N=16).
Study PIPa- or PIMb-related outcomes
Changes in PIP or PIM drugs Changes in specific PIP or PIM drugs

Tamblyn et al Number of PIP started per 1000 visits Cc: 52.2 Number of PIP started per 1000 visits: drug-disease contraindication C: 18.4
[30] d e f vs I: 16.6, RR 0.89 (CI 95% 0.72-1.10); drug-age contraindication C: 13.7 vs
vs I : 43.8, RR 0.82 (CI 95% 0.69 −0.98); PIP
I: 10.7, RR 0.77 (CI 95% 0.59-1.00); excessive duration therapy C: 17.1 vs I:
discontinuation C: 44.5% vs I: 47.5%, RR: 1.14
13.3, RR 0.78 (CI 95% 0.61-0.99); therapeutic duplication C: 6.8 vs I: 6.1, RR
(95% CI 0.98-1.33); number of PIP discontinued
0.87 (CI 95% 0.69-1.11); number of PIP discontinued per 1000 visits: drug-
per 1000 visits C: 67.4 vs I: 71.4, RR 1.06 (95%
disease contraindication C: 57.9 vs I: 62.6, RR 1.08 (CI 95% 0.85-1.36); drug-
CI 0.89-1.26)
age contraindication C: 42.9 vs I: 40.7, RR 0.94 (CI 95% 0.79-1.13); excessive
duration therapy C: 32.6 vs I: 32.3, RR 1.00 (CI 95% 0.77-1.29); therapeutic
duplication C: 334.0 vs I: 317.1, RR 0.94 (CI 95% 0.59-1.51)
Price et al [31] Change in PIP C: 0.1% vs I: 0.1%, P=.80
Avery et al —g At 6 months: history of peptic ulcer prescribed an NSAIDh without a PPI/his-
[41]
tory of peptic ulcer without PPIi AORj 0.58 (95% CI 0.38-0.89); asthma pre-
scribed a β blocker/asthma AOR 0.73 (95% CI 0.58-0.91); aged ≥75 years
long-term ACEk inhibitors or loop diuretics without urea and electrolyte
monitoring in the previous 15 months aged ≥75 years receiving long-term ACE
inhibitors or diuretics AOR 0.51 (95% CI 0.34-0.78); secondary outcomes
AOR varied from 0.39-0.96; at 12 months: history of peptic ulcer prescribed
an NSAID without a PPI/history of peptic ulcer without PPI AOR 0.91 (95%
CI 0.59-1.39); asthma prescribed a β blocker/asthma AOR 0.78 (95% CI 0.63-
0.97); aged ≥75 years receiving long-term ACE inhibitors or loop diuretics
without urea and electrolyte monitoring in the previous 15 months aged ≥75
years receiving long-term ACE inhibitors or diuretics AOR 0.63 (95% CI 0.41-
0.95); secondary outcomes AOR varied from 0.50-0.98
Erler et al [42] CKDl patients with ≥1 prescription exceeding NS differences in the numbers of patients with potentially dangerous or con-
recommended maximum dose AOR 0.46 (95% traindicated medications
CI 0.26-0.82); CKD patients with ≥1 prescription
exceeding recommended standard dose by >30%
AOR 0.66 (95% CI 0.36-1.21)
Clyne et al Percentage of PIP I: 52% vs C: 77%, P=.02, Odds of PIP AOR 0.30 (95% CI 0.14-0.68); NS differences for duplicate or
[43] AOR 0.32 (95% CI 0.15-0.70); mean number of long-term benzodiazepines
PIP C: 1.18 vs I: 0.70, P=.02
Cossette et al Drug cessation or dosage decrease: at 48h C: —
[40] 15.9% vs 45.8%, ADm 30.0% (95% CI 13.8-
46.1); at discharge C: 27.3% vs I: 48.1%, AD
20.8% (95% CI 4.6-37.0); drug cessation: at 48h
C: 15.1% vs 51.9%, AD 36.8% (95% CI 15.6-
57.9); at discharge C: 34.4% vs I: 45.2%, AD
10.7% (95% CI −10.5 to 31.9); dosage decrease:
at 48h C: 17.2% vs 38.1%, AD 20.9% (95% CI
4.1-45.8); at discharge C: 15.8% vs I: 52.4%,
AD 36.6% (95% CI 12.3-60.9)
Fried et al Proportion of medication reconciliation errors —
[32] corrected C: 14.3% vs I: 48.4%, P<.001; propor-
tion of ≥1 TRIM recommendations implemented
C: 21.9% vs I: 29.7%, P=.42
O’Sullivan et Patients with ≥1 PIP C: 84.6% vs I: 82% —
al [44]
Terrel et al Proportion of visits with a PIP C: 3.9% vs I: 2.6, —
[33] P=.02, ORn 0.55 (95% CI 0.34-0.89), ARRo
1.3% (95% CI 0.4-2.3); proportion of all pre-
scribed medications that were PIP C: 5.4% vs I:
3.4, P=.006, OR 0.59 (CI 95% 0.41-0.85), ARR
2.0% (95% CI 0.7-3.3)

XSL• FO
RenderX

Raebel et al Newly dispensed ≥1 PIP rate per 100 patients Newly dispensed ≥1 PIP rate per 100 patients: amitriptyline C: 0.61 vs I: 0.38,
[34] C: 2.20 vs I:1.85, P=.002, RRRp 16%; newly P<.001; chlordiazepoxide C: 0.05 vs I: 0.04, P=.55; diazepam C: 1.38 vs I:
dispensed ≥1 PIP only for indications included 1.28, P=.32; doxepin C: 0.14 vs I: 0.11, P=.24; flurazepam C: 0.01 vs I: 0.01,
in intervention rate per 100 patients C:1.50 vs P=.69; ketorolac C: 0.00 vs I: 0.01, P=.50; meperidine (oral) C: 0.01 vs I: 0.01,
I: 1.10, P<.001 P=N/Aq; oxycodone/aspirin C: 0.00 vs I: 0.00, P=N/A; newly dispensed ≥1
PIP only for indications included in intervention, rate per 100 patients:
amitriptyline C: 0.59 vs I: 0.37, P<.001; chlordiazepoxide C: 0.05 vs I: 0.04,
P=.55; diazepam C: 0.71 vs I: 0.56, P=.002; doxepin C: 0.13 vs I: 0.09, P=.17;
flurazepam C: 0.01 vs I: 0.01, P=.69; ketorolac C: 0.00 vs I: 0.01, P=.50;
meperidine (oral) C: 0.01 vs I: 0.01, P=N/A; oxycodone/aspirin C: 0.00 vs I:
0.00, P=N/A; dispensings of chlorpropamide, hydrocodone/aspirin, or piroxicam
C: 0 vs I: 0
Crossover studies
Peterson et al Prescription recommended daily dose C: 19% Prescription orders with 10-fold dosing: benzodiazepines C: 3.5% vs I: 2.0%,
[35] vs I: 29%, P<.001; prescription orders with 10- P=.01; opiates C: 5.5% vs I: 2.8%, P<.001; neuroleptics C: 10.0% vs I: 7.5%,
fold dosing C: 5.0% vs I: 2.8%, P<.001; prescrip- P=.35; prescriptions in agreement with recommendation: benzodiazepines C:
tions in agreement with recommendation C: 20.8% vs I: 28.2%, P<.001; opiates C: 16.6% vs I: 29%, P<.001; neuroleptics
18.6% vs I: 29.3%, P<.001; prescription of C: 22.5% vs I: 38%, P<.001
nonrecommended drugs C: 10.8% vs I: 7.6%,
P<.001
Ruhland et al — Glyburide orders from total oral antidiabetic orders Br: 3.3% vs As: 1.6%,
[36] P<.001; 17.8% patients transitioned off glyburide
Mattison et al Number of orders per total number of patients —
[37] per day: not recommended medication B: 0.070
vs A: 0.054, P<.001; dose reduction medications
B: 0.037 vs A: 0.037, P=.71; unflagged medica-
tions B: 0.033 vs A: 0.030, P=.03; number of
orders per number of new patients per day: not
recommended medication B: .333 vs A: 0.263,
P<.001; dose reduction medications B: 0.182 vs
A: 0.186, P=.51; unflagged medications B: 0.158
vs A: 0.148, P=.08
Lester et al — >65 years prescription rates of: diphenhydramine B: 26.9% vs A: 20%, P<.001;
[38] metoclopramide B: 16.7% vs A: 12.5%, P<.001; antipsychotics B: 8.8% vs A:
9.2%, P=.80; ≥65 years: no significant changes for diphenhydramine, metoclo-
pramide, or antipsychotics
Ghibelli et al Proportion of patients exposed to PIM at dis- Proportion of patients exposed to PIM at discharge: high-dose short-acting
[45] charge B: 37.8% vs A: 11.6%; mean number of benzodiazepines B: 21.6% vs A: 6.7%; ticlopidine B: 5.4% vs A: 0.0%;
PIM per patient at discharge B: 0.4 vs A: 0.1 digoxin B: 5. 4% vs A: 1.7%; doxazosin B: 1.3% vs A: 1.7%; clonidine B:
1.3% vs A: 0.0%

XSL• FO
RenderX

Stevens et al Average percentage of PIMs per month: site 1 —

[39] B: 11.9 vs A: 5.1, P<.001; site 2 B: 8.2 vs A:
4.5, P<.001; site 3 B: 8.9 vs A: 6.1, P=.007; site
4 B: 7.4 vs A: 5.7, P=.04
a
PIP: potentially inappropriate prescription.
b
PIM: potentially inappropriate medication.
c
d
e
RR: relative rate.
f
CI: confidence interval.
g
No data.
h
NSAID: nonsteroidal anti-inflammatory drug.
i
PPI: proton-pump inhibitor.
j
AOR: adjusted odds ratio.
k
ACE: angiotensin-converting enzyme.
l
CKD: chronic kidney disease.
m
AD: absolute difference.
n
OR: odds ratio.
o
ARR: absolute risk reduction.
p
RRR: relative risk reduction.
q
N/A: not applicable.
r
B: before.
s
A: after.

XSL• FO
RenderX
Table 4. Results of the included studies including number of prescriptions, adverse drug reactions, and potential drug-drug interactions (N=16).
Study Overall number of Adverse drug reaction PDDIa Others
prescriptions
Tamblyn et —b — Number of PDDI start- Physicians with more computer problems downloaded
al [30] ed per 1000 visits Cc: information less often (r=−.31)
1.5 vs I: 1.6, RRd 1.12
(CIe 95% 0.68-1.87);
number of PPDI discon-
tinued per 1000 visits
C: 68.6 vs If: 51.5 per
1000 visits, RR 1.33
(CI 95% 0.90-1.95)
Price et al — — — Description of 12 data quality probes; alert awareness:
[31] all participants in I were aware of STOPPg alerts, but
not consistently; workflow and display: location on
screen and workflow identified as barriers; study disrup-
tiveness: considered as minimal
Avery et al — — — Mean ICERh of intervention: at 6 months ₤65.6 (2.5-
[41] 97.5 percentile 58.2-73.0); at 12 months ₤66.5 (2.5-97.5
percentile 66.8-81.5)
Erler et al — — — —
[42]
Clyne et al — — — Beliefs about Medicine Questionnaire AORi 0.16 (CI
[43] 95% −1.85 to 1.07); 12-item Well-Being Questionnaire
AOR −0.41 (95% CI −0.80 to 1.07)
Cossette et — — — LOSj (median, IQRk) C: 9.5 (5-21) vs I: 10 (6-19), P=.9;
al [40] in-hospital death C:11 (8.6%) vs I: 6 (4.8%), P=.3; 30-
day post discharge ER visits C: 27 (21.1%) vs I: 27
(21.4%); 30-day postdischarge readmissions C: 28
(21.9%) vs I: 20 (15.9%), P=.3
Fried et al Mean number of — — Mean patient active participation C: 2.7 vs I: 5.5,
[32] medications per pa- P=.001; percentage of patients assessment of care for
tient C: 13.8 vs I: chronic conditions score >10 C: 15.6% vs I: 29.7%,
13.3, P=.65 P=.06, ORl 2.73 (CI 95% 0.82-9.08); patient medication-
related; communication C: 3.6 vs I: 7.5, P<.001; mean
clinician facilitative communication C: 0.67 vs I: 1.53,
P=.02; mean clinician medication-related communica-
tion C: 4.6 vs I:7.3, P=.002; percentage >1 recommen-
dations C: 32.8% vs I: 63.6%, P<.001; OR 3.33 (95%
CI 1.37-8.04)
O’Sullivan Total number of Patients with ≥1 ADRm — CDSq alerts 1000 in 296/361 patients; intervention group
et al [44] medications C: 3747 C: 20.7% vs I: 13.9%, attended 54.8% of recommendations; median (IQR)
vs I: 4192, P<.001;
P= 0.02, ARRn 6.8% LOS days C: 9 (5-16) vs I: 8 (5-13.5), P=.44; hospital
median (IQR) num- mortality C: 4.5% vs I: 4.7%, P>.05; interrater reliability
(95% CI 1.5-12.3);
ber of medications
RRRo 33.3% (95% CI; for application of WHO-UMCr ADR causality criteria
per patient C: 9 (7-
k= 0.81; Hallas ADR preventability criteria k= 0.87;
12) vs 12 (8-15), 7.7-51.7); NNTp 15
application of Hartwig ADR; severity criteria k=0.56
P<.001; number (%) (95% CI 8-68)
of people with
polypharmacy (≥5
medications); C: 346
(92.0) vs I: 346
(95.8), P=.44
Terrel et al — — — CDS alerts 114 during 107 visits; 43% of recommenda-
[33] tions accepted
Raebel et al — — — —
[34]
Crossover studies — — —

XSL• FO
RenderX
Study Overall number of Adverse drug reaction PDDIa Others

prescriptions
Peterson et Median (IQR) orders — — Number of altered mental status per 100 patient-days
al [35] per admission C: 2 C: 21.9 vs I: 20.9, P=.17; median (IQR) LOS days C:
(1-3) vs I: 4 2 (1-3), 4 (2-6) vs I: 4 (2-6), P=.43; in-hospital fall rate C: 0.64
P=.43 vs I: 0.28; falls per 100 patient-days, P<.001, AOR 0.50
(95% CI 0.30-0.82); fall injuries per 100 patient-days
rate C: 0.17 vs I: 0.06, P=.09
Pre-post intervention studies — — —
Ruhland et — — — CDS tool alerted 101 times for 75 providers during en-
al [36] counters for 76 patients over 90 days; physicians were
more likely to transition patients off glyburide vs other
health care providers (46.2% vs 8.0%, P<.001)
Mattison et — — — —
al [37]
Lester et al — — — —
[38]
Ghibelli et al — — Proportion of patients Median anticholinergic burden at discharge B: 1.5 vs
[45] exposed to PDDI at A: 1.1
discharge Bs: 87.8% vs
At: 88.3%; mean num-
ber of PDDI per patient
at discharge B: 4.5 vs
A: 3.7
Stevens et al — — — —
[39]
a
PDDI: potential drug-drug interactions.
b
No data.
c
d
RR: relative rate.
e
CI: confidence interval.
f
g
STOPP: Screening Tool of Older People’s Prescriptions.
h
ICER: incremental cost-effectiveness ratio.
i
AOR: adjusted odds ratio.
j
LOS: length of stay.
k
IQR: interquartile range.
l
OR: odds ratio.
m
ADR: adverse drug reaction.
n
ARR: absolute risk reduction.
o
RRR: relative risk reduction.
p
NNT: number needed to treat.
q
CDS: computerized decision support.
r
UMC: Uppsala Monitoring Centre.
s
B: before.
t
A: after.
Number of Prescriptions
Effects of Interventions
With regard to the impact on the number of prescriptions, the
The CDS tools consistently reduced the number of PIPs started
RCT described by Fried et al [32] reported no significant
and the mean number of PIPs per patient, while also increasing
reduction in the mean number of prescriptions in the group
PIM discontinuation and drug appropriateness. However, in
exposed to two Web apps. One study obtained information on
several cases statistical significance was not achieved for some
medications and chronic conditions from an electronic health
of the assessed measures, such as for PIM discontinuation in
record, and the second study used an interface for data chart
the Tamblyn et al article [30], for change in PIMs in the Price
review, a telephone-based patient assessment, a set of automated
et al study [31], and other studies described in Table 3.
algorithms evaluating medication appropriateness, and a

XSL• FO
RenderX
patient-specific medication management feedback report for Adherence to Computerized Decision Support Tools
the clinician. In a crossover study [35], there were no significant Several RCTs reported the frequency of adherence to CDS
differences in the median number of medications prescribed per recommendations by a health professional, with values ranging
patient during the periods in which guided dosing of from 33% to 55% [32,33,44]. No significant reduction in the
psychotropic medication was integrated into the Brigham length of stay or intrahospital mortality was found in the RCT
Integrated Computer System. described by O’Sullivan et al [44]; in the Cosstte et al study
In contrast, the RCT described by O’Sullivan et al [44] [40], the differences between the intervention and control groups
demonstrated that those in the intervention group (using CDS were not statistically different. Similarly, a crossover study
software structuring pharmacist review of medications designed found no difference in the length of stay between periods when
to optimize geriatric pharmaceutical care) prescribed the CDS tool was either active or inactive [35]. Likewise, no
significantly fewer drugs (both total and median number of difference was observed with respect to patients’ altered mental
drugs). However, no impact was observed for the proportion of status or fall injuries. However, there was a significant decrease
people with polypharmacy prescribed more than five drugs at in the in-hospital rate.
once. This RCT was the only one addressing adverse drug The TRIM RCT concluded that the use of CDS tools
reactions and it concluded that using this software significantly significantly improved patients’ active participation and
reduced the risk of adverse drug reactions. Furthermore, only facilitated communication between the clinician and the patient
15 patients’ medications needed to be reviewed to prevent one [32]. Another RCT found no significant impact on the Beliefs
adverse drug reaction. about Medicine Questionnaire or the 12-item Well-Being
Number of Potential Drug-Drug Interaction Questionnaire when general practitioners had access to
information from a pharmacist and a medical review with
Only two studies assessed whether CDS tools could decrease
Web-based pharmaceutical treatment algorithms and leaflets in
the number of potential drug-drug interactions [30,44]. One
addition to the usual care and simple, patient-level PIP postal
CDS used in an RCT was found to decrease the initiation of
feedback [43].
PIP, but it did not have a similar impact on deprescription [30].
One pre-post intervention study observed that the proportion
Cost-Effectiveness of Computerized Decision Support
of patients exposed to potential drug-drug interactions increased Tools
after implementing a computer-based app that collects, stores, The cost-effectiveness of CDS tools was addressed in one RCT.
and automatically provides drug information to reduce or The authors reported that there was a 95% probability that
prevent PIPs [45]. However, the mean number of potential adding a pharmacist-led information technology complex
drug-drug interactions per patient at discharge was reduced. intervention, in addition to computer-generated simple feedback,
Statistical significance was not reported. could be cost-effective, resulting in a willingness to pay ₤75
per error avoided at 6 months [41].
Other Measures
Other miscellaneous measures were reported in the studies Risk of Bias in the Studies Examined
examined, which should be highlighted. One RCT concluded The RCTs received a total score according to the Cochrane
that having computer problems was directly linked with PIP or Collaboration Risk of Bias tool that ranged from 1 [30,31] to 5
PIM information download, and these computer problems could [41,43]. The procedure to guarantee allocation concealment was
have an impact on the success of CDS tools [30]. Only one unclear in eight of ten RCTs. Complete blinding of participants
study described data quality probes; it found that professionals and personnel was not possible due to the nature of the
included in the intervention group were aware of STOPP alerts, intervention. Blinding for the outcome assessment was not
although not in a consistent manner. Furthermore, the layout conducted in five studies [31,34,40,41,44], and was unclear if
and impact on the workflow of the CDS tool were potential it was successful in another two [30,42]. Both of these biases
barriers to successful adherence [31]. may have resulted in an overestimate of the CDS tools’ impact
on PIP or PIM reduction (see Table 5).

XSL• FO
RenderX
Table 5. Risk of bias assessment (according to Cochrane Collaboration Risk of Bias tool) for the randomized controlled trials (n=10).
Study Risk of bias items Total score
(max=7)
Random se- Allocation Blinding of par- Blinding of out- Incomplete out- Selective Other
quence genera- concealment ticipants and come assessment come data reporting bias
tion personnel
Tamblyn et al [30] ?a ? –b ? ? +c – 1
Price et al [31] + ? – – ? ? – 1
Avery et al [41] + + – – + + + 5
Erler et al [42] + ? – ? + + – 3
Clyne et al [43] + ? – + + + + 5
Cossette et al [40] + ? – – – - + 2
Fried et al [32] – – – + + + ? 3
O’Sullivan et al [44] ? ? – – + + – 2
Terrel et al [33] + ? – + ? + – 3
Raebel et al [34] + ? – – ? + + 3
a
?: unclear risk of bias.
b
–: high risk of bias.
c
+: Low risk of bias.
Several studies did not report whether outcome data were the deprescribing process. Health professionals may have been
available for all the participants included (n=4) [30,31,33,34]. more susceptible to accepting the CDS tool recommendations.
Other biases were also found in five of the RCTs; namely, Alternatively, patients may have been more likely to agree with
selection bias, performance bias, contamination, and the withdrawal process. If a break in allocation concealment
underpowered sample sizes. occurred, it is expected that investigators may have potentially
included older adults that they considered best suited for the
Regarding the pre-post intervention studies [36-39,45], they
intervention group. Both types of bias may have led to an
were considered high risk following the Cochrane Effective
overestimation of the benefit of CDS tools.
Practice and Organisation of Care [46]. For example, it is
expected that pre-post intervention studies are more prone to We have also included five pre-post intervention studies. The
the Hawthorne effect [47]. The Hawthorne effect happens when nonrandomized nature of these studies is the major limitation
people (in this case, prescribers and patients) know they are of this analysis. The impact of CDS tools may be confounded
being watched, which may lead to changes in behavior [47]. by other changes that may have occurred in the institutions
We consider that it is possible that being aware of one’s study during the study periods.
participation could have resulted in prescribers taking more care
We observed that almost two-thirds of the included studies were
when prescribing medications.
performed in the United States, and one-third were performed
Limited generalizability was also pointed out by several authors in European countries. This reflects the importance that has
as a major limitation due to the context—single-center been given to this topic only in developed countries where
design—and the use of CDS tools that were created specifically electronic health record systems are widely available.
for the study, which may not be available in other institutions.
Overall Applicability and Quality of the Evidence
Discussion Seven studies were conducted in teaching hospitals and clinics
[33,36-38,40,44,45], which may indicate potential bias.
Principal Results Teaching units are more prone to accept interventions in patient
Despite the fact that withdrawal of PIPs is considered to be care, such as changes in a prescription through the use of CDS
evidence-based [48], it is not an easy task [49]. CDS tools may tools. We can assume that these professionals may be more
play a role in supporting deprescription. From the 16 studies likely to change a patient’s prescription and, therefore, to address
examined in this review, 10 were RCTs. Although RCTs PIPs. This tendency may result in an overestimate of the impact
represent stronger evidence, they lacked important data of the intervention, and we can only speculate as to what would
pertaining to clinical outcomes and presented a significant risk be the impact in a nonteaching unit.
of bias (the total score of the studies using the Cochrane There is a balance between the number of studies conducted in
Collaboration Risk of Bias tool ranged from 1 to 5 with a mean primary care versus secondary care institutions, and only one
value of 3). The most frequent biases included no blinding of was conducted in both. The impact of CDS on PIP or PIM
health professionals and an unclear risk of breaking allocation reduction was similar between settings despite differences in
concealment. If prescribers are not blinded, this can easily affect the health professional and population characteristics. This

XSL• FO
RenderX
suggests that the CDS tool might be successful in the context Strengths and Limitations
of a larger patient population. This review presents some limitations. We have chosen to
The generalization of our results may be limited for several include both RCTs (n=10) and pre-post studies (n=6). We
reasons. First, most studies used standard care as a comparator acknowledge that the latter provide a lower level of evidence.
without providing additional details. In such a complex context, Nevertheless, they have assessed some outcomes for which no
the management of older patients in institutions with several additional evidence exists. In addition, we have focused our
levels of care may mean that standard care could differ greatly search on articles having PIP modification outcomes, thus some
between studies. studies assessing changes in PIM may have been missed.
Second, the intervention varied greatly as a result of using Our search terms were more limited to PIP; therefore, this paper
different electronic systems, contents, and layouts. The may have missed some studies regarding PIM. Nevertheless,
intervention frequently included several features beyond the no new articles were found when searching in the references
creation and application of a CDS tool itself. from the included studies and in the grey literature
Third, the main outcome definition was also diverse. Several Major strengths of our study include the fact that we have
studies used STOPP [31,32,40] and Beers Criteria followed the Cochrane Collaboration Handbook [50], which
[32,34,39,40,45] to define which medications were targeted. makes our study less susceptible to major biases and errors.
Both criteria are widely used worldwide, and although they do Furthermore, no new references were found from searches in
not provide a list of prohibited medications, they are an the grey literature, pertinent scientific meeting books of
important tool for physicians due to their evidence-based abstracts, and the included studies’ list of references, which
rationale and constant updating. Nevertheless, the authors chose suggests that our search strategy was exhaustive and all pertinent
different groups of criteria for their outcome measures. articles had been included.
Fourth, the studies selected different participants and had widely However, the quality of the results of a systematic review is
variable sample sizes. Only two studies addressed potential dependent on the available data. For all that was previously
drug-drug interactions [30,45] and one addressed adverse drug described, we believed that conducting a meta-analysis was not
reactions [44]. Due to the increase of polypharmacy in older possible. Thus, only a narrative synthesis has been provided.
adults, the risk is higher for experiencing drug-drug interactions
Comparison With Prior Work
and adverse drug reactions. For the former, no significant impact
was found, whereas for the latter, using a CDS tool significantly To our knowledge, there are three previously published
decreased the number of adverse drug reactions. systematic reviews assessing the impact of CDS tools on PIP
or PIM [51-27]. Due to an increase in the search period, the use
This tool, which included a clinical decision support software of broader search criteria, and our overall methodology, we
and a structured pharmacist review of medication [44], seems were able to include five additional RCTs [31,32,40,43,44].
to be promising for aiding medication reconciliation activities. These studies added evidence with new outcomes, such as
Most of the reconciliation issues highlighted by this CDS tool well-being and patients’ beliefs [43], reduction of adverse drug
were accepted by the health care professionals involved. In reactions [44], and users’ perspectives [31].
particular, the Erler et al study [42] should, in our opinion, have
assessed these two topics because they studied a population The highlight of the findings in the more recent RCTs were as
with renal impairment, which is particularly susceptible to follows. In the study by Price et al [31], alerts with specific
adverse drug reactions and drug interactions. Similarly, only STOPP guideline content in electronic medical records
two studies assessed the impact of CDS tools on length of stay positively changed PIPs (comparator: 0.1% versus intervention:
[35,40], and two assessed intrahospital mortality [40,44]. No 0.1%, P=.80), but not significantly. In the study by Clyne et al
differences were found between those using a CDS tool and [43], the intervention consisted of Web-based pharmaceutical
those not using a CDS tool. Cost-effectiveness was also assessed treatment algorithms that led to a lower percentage of PIPs
by one study, which reported a 95% probability of a CDS tool (intervention: 52% versus comparator: 77%, P=.02). In the trial
being cost-effective due to a willingness to pay ₤75 to prevent by Cossette et al [40], a computerized alert system-based
an adverse drug reaction in a 6-month period [41]. The study’s pharmacist-physician intervention was able to significantly
results may have been underestimated due to low adherence to increase drug cessation or decrease dosage at discharge
CDS recommendations. Three RCTs that evaluated adherence (comparator: 27.3% versus intervention: 48.1%; absolute
reported values fluctuating from 33% to 55% [32,33,44]. Finally, difference 20.8%, 95% CI 4.6-37.0). In the TRIM trial [32], the
we consider the possibility that the Avery et al trial [41] could proportion of medication reconciliation errors was significantly
have explored the issue of prescription NSAIDs to patients with diminished (comparator: 14.3% versus intervention: 48.4%,
a history of asthma as a secondary outcome because the authors P<.001). In the article by O’Sullivan et al [44], clinical decision
had information on both conditions (prescriptions of NSAIDs support software reduced adverse drug reactions among older
and a history of asthma). This analysis could yield interesting patients (control patients: 20.7% versus intervention patients:
information about the patterns of prescribing NSAIDs to these 13.9%, P=.02). In sum, articles published since 2012
patients. substantiated the value of CDS to improve PIP- or PIM-related
outcomes.

XSL• FO
RenderX
Conclusions This research suggests that RCTs assessing the impact of CDS
The use of CDS tools had a positive impact on PIP tools could be conducted in both primary and secondary health
independently of the outcome definition in the majority of the care settings using medication targets defined by Beers or
studies included in our analysis. However, statistical significance STOPP criteria.
was not always achieved. Several possible sources of bias and To replicate the intervention in different RCTs, a standard CDS
experimental limitations were found in the included studies, tool could be developed. These CDS tools could promote
and evidence is lacking regarding the impact of CDS tools in communication between physicians and pharmaceutical servives.
potential drug-drug interactions, adverse drug reactions, length These RCTs could also assess adverse drug reactions, quality
of stay, mortality, and cost-effectiveness. of life measurements, and patient and professional satisfaction,
with a reasonable follow-up to clarify the clinical usefulness of
these tools.
Acknowledgments
This article was supported by National Funds through FCT-Fundação para a Ciência e a Tecnologia within CINTESIS, R&D
Unit (reference UID/IC/4255/2019).
Conflicts of Interest
None declared.
References
1. OECD Data. 2019. Elderly population indicator URL: https://data.oecd.org/pop/elderly-population.htm#indicator-chart
[accessed 2019-05-20]
2. World Health Organization. 2019. Ageing and health URL: https://www.who.int/news-room/fact-sheets/detail/
ageing-and-health [accessed 2019-05-12]
3. World Health Organization. World Report On Ageing And Health. Geneva: World Health Organization; 2019.
4. World Health Organization. 2019. The global strategy and action plan on ageing and health URL: https://www.who.int/
ageing/global-strategy/en/ [accessed 2019-06-22]
5. Prazeres F, Santiago L. Prevalence of multimorbidity in the adult population attending primary care in Portugal: a
cross-sectional study. BMJ Open 2015 Sep 25;5(9):e009287 [FREE Full text] [doi: 10.1136/bmjopen-2015-009287]
[Medline: 26408832]
6. Formiga F, Ferrer A, Sanz H, Marengoni A, Alburquerque J, Pujol R, Octabaix study members. Patterns of comorbidity
and multimorbidity in the oldest old: the Octabaix study. Eur J Intern Med 2013 Jan;24(1):40-44. [doi:
10.1016/j.ejim.2012.11.003] [Medline: 23186603]
7. Lu W, Wen Y, Chen L, Hsiao F. Effect of polypharmacy, potentially inappropriate medications and anticholinergic burden
on clinical outcomes: a retrospective cohort study. CMAJ 2015 Mar 03;187(4):E130-E137 [FREE Full text] [doi:
10.1503/cmaj.141219] [Medline: 25646290]
8. Gnjidic D, Hilmer SN, Blyth FM, Naganathan V, Waite L, Seibel MJ, et al. Polypharmacy cutoff and outcomes: five or
more medicines were used to identify community-dwelling older men at risk of different adverse outcomes. J Clin Epidemiol
2012 Sep;65(9):989-995. [doi: 10.1016/j.jclinepi.2012.02.018] [Medline: 22742913]
9. Payne RA, Avery AJ, Duerden M, Saunders CL, Simpson CR, Abel GA. Prevalence of polypharmacy in a Scottish primary
care population. Eur J Clin Pharmacol 2014 May;70(5):575-581. [doi: 10.1007/s00228-013-1639-9] [Medline: 24487416]
10. Motter FR, Fritzen JS, Hilmer SN, Paniz ÉV, Paniz VM. Potentially inappropriate medication in the elderly: a systematic
review of validated explicit criteria. Eur J Clin Pharmacol 2018 Mar 27;74(6):679-700. [doi: 10.1007/s00228-018-2446-0]
11. Marengoni A, Angleman S, Melis R, Mangialasche F, Karp A, Garmen A, et al. Aging with multimorbidity: a systematic
review of the literature. Ageing Res Rev 2011 Sep;10(4):430-439. [doi: 10.1016/j.arr.2011.03.003] [Medline: 21402176]
12. Weng M, Tsai C, Sheu K, Lee Y, Lee H, Tzeng S, et al. The impact of number of drugs prescribed on the risk of potentially
inappropriate medication among outpatient older adults with chronic diseases. QJM 2013 Nov;106(11):1009-1015. [doi:
10.1093/qjmed/hct141] [Medline: 23836694]
13. Tommelein E, Mehuys E, Petrovic M, Somers A, Colin P, Boussery K. Potentially inappropriate prescribing in
community-dwelling older people across Europe: a systematic literature review. Eur J Clin Pharmacol 2015
Dec;71(12):1415-1427. [doi: 10.1007/s00228-015-1954-4] [Medline: 26407687]
14. Petrovic M, van der Cammen T, Onder G. Adverse drug reactions in older people: detection and prevention. Drugs Aging
2012 Jun 01;29(6):453-462. [doi: 10.2165/11631760-000000000-00000] [Medline: 22642780]
15. Cherubini A, Ruggiero C, Gasperini B, Dell'Aquila G, Cupido MG, Zampi E, et al. The prevention of adverse drug reactions
in older subjects. Curr Drug Metab 2011 Sep;12(7):652-657. [Medline: 21495975]

XSL• FO
RenderX
16. Reeve E, Gnjidic D, Long J, Hilmer S. A systematic review of the emerging deﬁnition of 'deprescribing' with network
analysis: implications for future research and clinical practice. Br J Clin Pharmacol 2015 Dec;80(6):1254-1268 [FREE Full
text] [doi: 10.1111/bcp.12732] [Medline: 27006985]
17. Haverhals LM, Lee CA, Siek KA, Darr CA, Linnebur SA, Ruscin JM, et al. Older adults with multi-morbidity: medication
management processes and design implications for personal health applications. J Med Internet Res 2011;13(2):e44 [FREE
Full text] [doi: 10.2196/jmir.1813] [Medline: 21715286]
18. Mira JJ, Navarro I, Botella F, Borrás F, Nuño-Solinís R, Orozco D, et al. A Spanish pillbox app for elderly patients taking
multiple medications: randomized controlled trial. J Med Internet Res 2014;16(4):e99 [FREE Full text] [doi:
10.2196/jmir.3269] [Medline: 24705022]
19. Clyne B, Fitzgerald C, Quinlan A, Hardy C, Galvin R, Fahey T, et al. Interventions to address potentially inappropriate
prescribing in community-dwelling older adults: a systematic review of randomized controlled trials. J Am Geriatr Soc
2016 Jun;64(6):1210-1222. [doi: 10.1111/jgs.14133] [Medline: 27321600]
20. Cooper JA, Cadogan CA, Patterson SM, Kerse N, Bradley MC, Ryan C, et al. Interventions to improve the appropriate use
of polypharmacy in older people: a Cochrane systematic review. BMJ Open 2015 Dec 09;5(12):e009235 [FREE Full text]
[doi: 10.1136/bmjopen-2015-009235] [Medline: 26656020]
21. Loganathan M, Singh S, Franklin BD, Bottle A, Majeed A. Interventions to optimise prescribing in care homes: systematic
review. Age Ageing 2011 Mar;40(2):150-162. [doi: 10.1093/ageing/afq161] [Medline: 21262782]
22. Rankin A, Cadogan CA, Patterson SM, Kerse N, Cardwell CR, Bradley MC, et al. Interventions to improve the appropriate
use of polypharmacy for older people. Cochrane Database Syst Rev 2018 Dec 03;9:CD008165. [doi:
10.1002/14651858.CD008165.pub4] [Medline: 30175841]
23. Gnjidic D, Le Couteur DG, Kouladjian L, Hilmer SN. Deprescribing trials: methods to reduce polypharmacy and the impact
on prescribing and clinical outcomes. Clin Geriatr Med 2012 May;28(2):237-253. [doi: 10.1016/j.cger.2012.01.006]
[Medline: 22500541]
24. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision
support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10(6):523-530 [FREE
Full text] [doi: 10.1197/jamia.M1370] [Medline: 12925543]
25. Payne TH. Computer decision support systems. Chest 2000 Aug;118(2 Suppl):47S-52S. [doi: 10.1378/chest.118.2_suppl.47s]
[Medline: 10939999]
26. Martins CM, da Costa Teixeira AS, de Azevedo LF, Sá LM, Santos PA, do Couto ML, et al. The effect of a test ordering
software intervention on the prescription of unnecessary laboratory tests-a randomized controlled trial. BMC Med Inform
Decis Mak 2017 Feb 20;17(1):20 [FREE Full text] [doi: 10.1186/s12911-017-0416-6] [Medline: 28219437]
27. Clyne B, Bradley MC, Hughes C, Fahey T, Lapane KL. Electronic prescribing and other forms of technology to reduce
inappropriate medication use and polypharmacy in older people: a review of current evidence. Clin Geriatr Med 2012
May;28(2):301-322. [doi: 10.1016/j.cger.2012.01.009] [Medline: 22500545]
28. Monteiro L, Maricoto T, Solha IS, Monteiro-Soares M, Martins C. Computerised decision to reduce inappropriate medication
in the elderly: a systematic review with meta-analysis protocol. BMJ Open 2018 Jan 30;8(1):e018988. [doi:
10.1136/bmjopen-2017-018988]
29. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing
risk of bias in randomised trials. BMJ 2011 Oct 18;343(oct18 2):d5928-d5928. [doi: 10.1136/bmj.d5928]
30. Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, et al. The medical office of the 21st century (MOXXI):
effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ 2003
Sep 16;169(6):549-556 [FREE Full text] [Medline: 12975221]
31. Price M, Davies I, Rusk R, Lesperance M, Weber J. Applying STOPP guidelines in primary care through electronic medical
record decision support: randomized control trial highlighting the importance of data quality. JMIR Med Inform 2017 Jun
15;5(2):e15 [FREE Full text] [doi: 10.2196/medinform.6226] [Medline: 28619704]
32. Fried TR, Niehoff KM, Street RL, Charpentier PA, Rajeevan N, Miller PL, et al. Effect of the tool to reduce inappropriate
medications on medication communication and deprescribing. J Am Geriatr Soc 2017 Oct;65(10):2265-2271 [FREE Full
text] [doi: 10.1111/jgs.15042] [Medline: 28804870]
33. Terrell KM, Perkins AJ, Dexter PR, Hui SL, Callahan CM, Miller DK. Computerized decision support to reduce potentially
inappropriate prescribing to older emergency department patients: a randomized, controlled trial. J Am Geriatr Soc 2009
Aug;57(8):1388-1394. [doi: 10.1111/j.1532-5415.2009.02352.x] [Medline: 19549022]
34. Raebel MA, Charles J, Dugan J, Carroll NM, Korner EJ, Brand DW, et al. Randomized trial to improve prescribing safety
in ambulatory elderly patients. J Am Geriatr Soc 2007 Jul;55(7):977-985. [doi: 10.1111/j.1532-5415.2007.01202.x] [Medline:
17608868]
35. Peterson JF, Kuperman GJ, Shek C, Patel M, Avorn J, Bates DW. Guided prescription of psychotropic medications for
geriatric inpatients. Arch Intern Med 2005 Apr 11;165(7):802-807. [doi: 10.1001/archinte.165.7.802] [Medline: 15824302]
36. Ruhland DJ, Bellone JM, Wilkes E. Implementation and assessment of an ambulatory prescribing guidance tool to improve
patient safety in the geriatric population. Consult Pharm 2017 Mar 01;32(3):169-174. [doi: 10.4140/TCP.n.2017.169]
[Medline: 28270272]

XSL• FO
RenderX
37. Mattison ML, Afonso KA, Ngo LH, Mukamal KJ. Preventing potentially inappropriate medication use in hospitalized older
patients with a computerized provider order entry warning system. Arch Intern Med 2010 Aug 09;170(15):1331-1336
[FREE Full text] [doi: 10.1001/archinternmed.2010.244] [Medline: 20696957]
38. Lester PE, Rios-Rojas L, Islam S, Fazzari MJ, Gomolin IH. Impact of computerized physician order entry alerts on prescribing
in older patients. Drugs Aging 2015 Mar;32(3):227-233. [doi: 10.1007/s40266-015-0244-2] [Medline: 25752906]
39. Stevens M, Hastings SN, Markland AD, Hwang U, Hung W, Vandenberg AE, et al. Enhancing Quality of Provider Practices
for Older Adults in the Emergency Department (EQUiPPED). J Am Geriatr Soc 2017 Jul;65(7):1609-1614. [doi:
10.1111/jgs.14890] [Medline: 28388818]
40. Cossette B, Éthier J, Joly-Mischlich T, Bergeron J, Ricard G, Brazeau S, et al. Reduction in targeted potentially inappropriate
medication use in elderly inpatients: a pragmatic randomized controlled trial. Eur J Clin Pharmacol 2017
Oct;73(10):1237-1245. [doi: 10.1007/s00228-017-2293-4] [Medline: 28717929]
41. Avery AJ, Rodgers S, Cantrill JA, Armstrong S, Cresswell K, Eden M, et al. A pharmacist-led information technology
intervention for medication errors (PINCER): a multicentre, cluster randomised, controlled trial and cost-effectiveness
analysis. Lancet 2012 Apr 07;379(9823):1310-1319 [FREE Full text] [doi: 10.1016/S0140-6736(11)61817-5] [Medline:
22357106]
42. Erler A, Beyer M, Petersen JJ, Saal K, Rath T, Rochon J, et al. How to improve drug dosing for patients with renal impairment
in primary care--a cluster-randomized controlled trial. BMC Fam Pract 2012 Sep 06;13:91 [FREE Full text] [doi:
10.1186/1471-2296-13-91] [Medline: 22953792]
43. Clyne B, Smith SM, Hughes CM, Boland F, Bradley MC, Cooper JA, OPTI-SCRIPT study team. Effectiveness of a
multifaceted intervention for potentially inappropriate prescribing in older patients in primary care: a cluster-randomized
controlled trial (OPTI-SCRIPT study). Ann Fam Med 2015 Nov;13(6):545-553 [FREE Full text] [doi: 10.1370/afm.1838]
[Medline: 26553894]
44. O'Sullivan D, O'Mahony D, O'Connor MN, Gallagher P, Gallagher J, Cullinan S, et al. Prevention of adverse drug reactions
in hospitalised older patients using a software-supported structured pharmacist intervention: a cluster randomised controlled
trial. Drugs Aging 2016 Jan;33(1):63-73. [doi: 10.1007/s40266-015-0329-y] [Medline: 26597401]
45. Ghibelli S, Marengoni A, Djade CD, Nobili A, Tettamanti M, Franchi C, et al. Prevention of inappropriate prescribing in
hospitalized older patients using a computerized prescription support system (INTERcheck(®)). Drugs Aging 2013
Oct;30(10):821-828. [doi: 10.1007/s40266-013-0109-5] [Medline: 23943248]
46. Cochrane Effective Practice and Organization of Care. EPOC resources for review authors URL: https://epoc.cochrane.org/
resources/epoc-resources-review-authors [accessed 2019-06-22]
47. Indrayan A. Medical Biostatistics. Third edition. New York: Chapman And Hall; 2013.
48. The 2019 American Geriatrics Society Beers Criteria® Update Expert Panel. American Geriatrics Society 2019 updated
AGS Beers Criteria® for potentially inappropriate medication use in older adults. J Am Geriatr Soc 2019 Apr;67(4):674-694.
[doi: 10.1111/jgs.15767] [Medline: 30693946]
49. Anderson K, Stowasser D, Freeman C, Scott I. Prescriber barriers and enablers to minimising potentially inappropriate
medications in adults: a systematic review and thematic synthesis . BMJ Open 2014 Dec 08;4(12):e006544 [FREE Full
text] [doi: 10.1136/bmjopen-2014-006544] [Medline: 25488097]
50. Cochrane Training. 2019. Cochrane handbook for systematic reviews of interventions URL: https://training.cochrane.org/
handbook [accessed 2019-06-22]
51. Clyne B, Fitzgerald C, Quinlan A, Hardy C, Galvin R, Fahey T, et al. Interventions to address potentially inappropriate
prescribing in community-dwelling older adults: a systematic review of randomized controlled trials. J Am Geriatr Soc
2016 Jun;64(6):1210-1222. [doi: 10.1111/jgs.14133] [Medline: 27321600]
52. Loganathan M, Singh S, Franklin BD, Bottle A, Majeed A. Interventions to optimise prescribing in care homes: systematic
review. Age Ageing 2011 Mar;40(2):150-162. [doi: 10.1093/ageing/afq161] [Medline: 21262782]
Abbreviations
CDS: computerized decision support
NSAID: nonsteroidal anti-inflammatory drugs
PDDI: potential drug-drug interaction
PIM: potentially inappropriate medications
PIP: potentially inappropriate prescriptions
RCT: randomized controlled trial

XSL• FO
RenderX
Edited by G Eysenbach; submitted 06.07.19; peer-reviewed by G Signorelli, B Alper, C Alvarez, C Kanu, M Basit; comments to author
01.08.19; revised version received 12.09.19; accepted 23.09.19; published 14.11.19
Please cite as:
Monteiro L, Maricoto T, Solha I, Ribeiro-Vaz I, Martins C, Monteiro-Soares M
Reducing Potentially Inappropriate Prescriptions for Older Patients Using Computerized Decision Support Tools: Systematic Review
J Med Internet Res 2019;21(11):e15385
URL: https://www.jmir.org/2019/11/e15385
doi: 10.2196/15385
PMID: 31724956
©Luís Monteiro, Tiago Maricoto, Isabel Solha, Inês Ribeiro-Vaz, Carlos Martins, Matilde Monteiro-Soares. Originally published
in the Journal of Medical Internet Research (http://www.jmir.org), 14.11.2019. This is an open-access article distributed under
the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted
use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet
Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/,
as well as this copyright and license information must be included.

XSL• FO
RenderX

(JURNAL, Eng) Reducing Potentially Inappropriate Prescriptions For Older Patients Using Computerized Decision Support Tools, Systematic Review

Uploaded by

Copyright:

Available Formats

(JURNAL, Eng) Reducing Potentially Inappropriate Prescriptions For Older Patients Using Computerized Decision Support Tools, Systematic Review

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

(JURNAL, Eng) Reducing Potentially Inappropriate Prescriptions For Older Patients Using Computerized Decision Support Tools, Systematic Review

Uploaded by

Copyright:

Available Formats

JOURNAL OF MEDICAL INTERNET RESEARCH Monteiro et al

Reducing Potentially Inappropriate Prescriptions for Older Patients

(J Med Internet Res 2019;21(11):e15385) doi: 10.2196/15385

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 1

have been published since then [27]. This systematic review

Due to changes in pharmacokinetics and pharmacodynamics, Selection Process

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 3

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 4

Table 1. Descriptions of the included studies in the systematic review (N=16).

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 5

Author, year; (study); country Setting Comparator Intervention Deprescribing target

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 6

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 7

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 8

Randomized controlled trials

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 9

Study PIPa- or PIMb-related outcomes

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 10

Study PIPa- or PIMb-related outcomes

Stevens et al Average percentage of PIMs per month: site 1 —

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 11

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 12

Study Overall number of Adverse drug reaction PDDIa Others

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 13

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 14

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 15

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 16

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 17

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 18

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 19

https://www.jmir.org/2019/11/e15385 J Med Internet Res 2019 | vol. 21 | iss. 11 | e15385 | p. 20

You might also like