Journal of the American Medical Informatics Association, 24(2), 2017, 413–422

doi: 10.1093/jamia/ocw145
Advance Access Publication Date: 7 October 2016
Review

Review

Impact of commercial computerized provider order entry

(CPOE) and clinical decision support systems (CDSSs)

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

on medication errors, length of stay, and mortality in
intensive care units: a systematic review and
meta-analysis
Mirela Prgomet,1 Ling Li,1 Zahra Niazkhani,2,3 Andrew Georgiou,1 and
Johanna I Westbrook1
1
Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia,
2
Department of Health Information Technology, Urmia University of Medical Sciences, Urmia, Iran and 3Nephrology and Kidney
Transplant Research Center, Urmia University of Medical Sciences, Urmia, Iran

Correspondence to Dr Mirela Prgomet, Centre for Health Systems and Safety Research, Australian Institute of Health
Innovation, Level 6, 75 Talavera Road, Macquarie University, Sydney NSW 2109, Australia; [email protected];
Fax: þ612 9850 2499.
Received 5 June 2016; Revised 9 August 2016; Accepted 31 August 2016

ABSTRACT
Objective: To conduct a systematic review and meta-analysis of the impact of commercial computerized pro-
vider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay
(LOS), and mortality in intensive care units (ICUs).
Methods: We searched for English-language literature published between January 2000 and January 2016 us-
ing Medline, Embase, and CINAHL. Titles and abstracts of 586 unique citations were screened. Studies were in-
cluded if they: (1) reported results for an ICU population; (2) evaluated the impact of CPOE or the addition of
CDSSs to an existing CPOE system; (3) reported quantitative data on medication errors, ICU LOS, hospital LOS,
ICU mortality, and/or hospital mortality; and (4) used a randomized controlled trial or quasi-experimental study
design.
Results: Twenty studies met our inclusion criteria. The transition from paper-based ordering to commercial
CPOE systems in ICUs was associated with an 85% reduction in medication prescribing error rates and a 12%
reduction in ICU mortality rates. Overall meta-analyses of LOS and hospital mortality did not demonstrate a sig-
nificant change.
Discussion and Conclusion: Critical care settings, both adult and pediatric, involve unique complexities, making
them vulnerable to medication errors and adverse patient outcomes. The currently limited evidence base re-
quires research that has sufficient statistical power to identify the true effect of CPOE implementation. There is
also a critical need to understand the nature of errors arising post-CPOE and how the addition of CDSSs can be
used to provide greater benefit to delivering safe and effective patient care.

Key words: medical order entry systems, decision support systems, clinical, medication errors, mortality, length of stay

C The Author 2016. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved.
V
For Permissions, please email: [email protected]
413
414 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2

INTRODUCTION Database Searches

The high rate of medication errors in hospitals is a well-recognized Medline Embase Cinahl
and significant patient safety issue.1 Medication errors have consis- n = 241 n = 447 n = 166
tently been attributed to longer hospital stays, increased costs, sig- Inventory of Evaluaon Health IT Bibliography
nificant morbidity, and even death.1–4 In complex hospital wards, Publicaons n = 16 n = 17
such as intensive care units (ICUs), the prevalence of errors and ad-
verse patient outcomes is higher and of greater severity than in gen-
Total citaons from database searches
eral wards.5 A clinical review of medical errors in critical care
n = 887
undertaken by Moyen et al.6 associated this increased prevalence of
errors to risk factors related to the severity of illness of ICU patients, Duplicate citaons removed
n = 301
number and type of medications used (i.e., frequent use of boluses
Unique citaons screened
and infusions, which often require weight-based dose calculations), n = 586 Exclusions based on tle and
and complexity of the ICU environment. abstract review

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

Interventions aimed at preventing medication errors in hospitals, n = 462
particularly at the prescribing stage, include computerized provider
Full-text assessed Exclusions based on full-text
order entry (CPOE) and clinical decision support systems (CDSSs). n = 124 n = 107
Systematic reviews of the impact of CPOE and CDSSs across inpa- Primary reasons for exclusion:
tient settings have reported significant reductions in medication er- • Intervenon (not CPOE, CDSS
rors,7–11 while changes in mortality10,12 and length of stay not connected to CPOE, not a
(LOS)11,12 have not been significant. However, these reviews com- commercial system) (n=43)
• Study type (qualitave, review,
bined results from homegrown and commercial CPOE systems.
proof of concept/system
Homegrown systems are more likely to demonstrate positive effects tesng, abstract/poster) (n=39)
on safety and quality of care, as they are under the local control of • Outcomes (no outcomes of
the implementing institution and have been highly customized for interest, not ICU or no ICU
specific outcomes) (n=25)
local conditions.9,13 As homegrown systems become increasingly
difficult for organizations to maintain, almost all future system im-
plementations are likely to involve commercial systems.13
Arcles added via hand-searching
The lack of specific reviews to guide the large investments being and reference list checking
made in sophisticated commercial systems highlights the need to col- Arcles included n=3
late and examine research that evaluates the impact of commercial n = 20
CPOE systems on errors and patient outcomes,7,8 particularly among
Figure 1. Search and selection flow-diagram.
populations most at risk of errors and adverse outcomes, such as pa-
tients in ICUs. Thus, our aim was to conduct a systematic review and
meta-analysis of evidence of the impact of commercial CPOE and
CDSSs on medication errors, LOS, and mortality in ICUs. duplicates, the titles and abstracts of 586 unique citations were
screened for eligibility. Screening of titles and abstracts was con-
ducted independently by 2 researchers and compared for consis-
METHOD tency. Where there was a discrepancy between researchers, the
citation was assigned to full-text review. Two researchers also inde-
Search strategy pendently reviewed the full text of 124 studies. The authors of 13
We searched for English-language literature published between Janu- studies were contacted in order to obtain clarification or additional
ary 2000 and January 2016 using Medline and Embase via Ovid, and data. Three researchers assessed the full text of a subset of 34 stud-
The Cumulative Index to Nursing and Allied Health Literature ies, which were discussed in depth against the eligibility criteria to
(CINAHL) via EBSCOhost. We restricted the date range, as studies determine the final set of included studies. Figure 1 shows the study
conducted prior to 2000 tended to evaluate homegrown CPOE.7 We selection process.
used a combination of MeSH terms and keywords related to the inter- We defined CPOE as computer-based systems used for entering
vention (CPOE, CDSS), outcomes of interest (medication errors, LOS, orders, including laboratory tests, imaging, nutrition, blood prod-
mortality), and study setting (ICU). The complete database search ucts, and medication prescriptions. Almost all CPOE systems have
strategy is provided in Appendix A (available as a Supplementary File). some level of decision support to assist ordering decisions; however,
We also searched 2 specialized bibliographies—the Inventory of Health the degree of sophistication of CDSSs can vary from basic duplicate
Information Evaluation Studies (https://evaldb.umit.at/) and the Health order alerts to complex algorithms based on patient-specific data.15
IT Bibliography (https://healthit.ahrq.gov/health-it-tools-and-resources/ Where studies evaluated the addition of a specific CDSS to an exist-
health-it-bibliography)—and hand searched the reference lists of all ing CPOE system, such as algorithms developed in response to iden-
full-text articles that we assessed for potential inclusion. The protocol tified medical errors or quality improvement initiatives, we defined
for this review was guided by the Preferred Reporting Items for Sys- these as “targeted” CDSSs.16 For studies reporting medication er-
tematic Reviews and Meta-Analyses (PRISMA) statement14 and was rors, we focused on errors occurring at the prescribing stage, which
registered with PROSPERO (registration number CRD42013004543). can include incomplete, incorrect, or inappropriate drug orders.
Studies were eligible for inclusion if they: (1) reported results for
Study selection an ICU population; (2) evaluated the impact of moving from paper-
The above search strategy was executed on 17 February 2016 and based ordering to CPOE or evaluated the addition of a targeted
resulted in the identification of 887 citations. After removing CDSS to an existing CPOE system; (3) reported quantitative data on
Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2 415

medication errors, LOS, or mortality pre- and post-CPOE or CDSS; The results from the longest study periods (i.e., 24-month baseline
and (4) used a randomized controlled trial or quasi-experimental and 12-month intervention periods) were used for the meta-
study design. analyses, while results from the other study periods were used for
Studies were excluded if the CPOE system was not a commercial sensitivity analysis. Three studies27–29 reported results from 2 sepa-
system, was implemented prior to the year 2000, or was imple- rate intervention periods (e.g., two 2-week periods). We combined
mented alongside other interventions making it difficult to assess the the data from the 2 periods into 1 intervention period (e.g., one 4-
impact of CPOE (e.g., Abstoss et al.17). For studies assessing the ad- week period). A study by Kadmon et al.30 on the impact of CPOE
dition of a CDSS to an existing CPOE system, if the CDSS was not on medication errors included 2 interventions: post-CPOE and post-
integrated with the CPOE system (e.g., Sintchenko et al.18), the CPOE with CDSS. We included the results from the latter in the
study was excluded. We also excluded studies where outcomes were meta-analysis and conducted sensitivity analysis for the results from
voluntarily reported (e.g., nurses reporting errors in incident report- the post-CPOE–only intervention period.
ing systems); studies that were conducted in a simulated environ- The included studies contained sufficient information to conduct
ment; qualitative studies or opinion pieces; and studies that were meta-analyses for 4 outcome measures: medication errors, ICU

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

available only as abstracts or posters, as they provided insufficient LOS, ICU mortality, and hospital mortality. We calculated relative
information to systematically determine eligibility. risks (RRs) for medication errors, ICU mortality, and hospital mor-
tality, and mean difference for ICU LOS. A meta-analysis for each
Data extraction outcome measure was performed using random effects models to
pool the results and, in order to be conservative, the Knapp-Hartung
We extracted the following data variables into a spreadsheet: study
approach31 was applied to account for heterogeneity between stud-
author(s), year of publication, country in which the study was con-
ies. The meta-analyses results are presented using forest plots (Fig-
ducted, type of ICU (adult/pediatric), study design, number and type
ures 2–5). Between-study heterogeneity was evaluated using
of patients included in the study (all ICU patients/only patients with
chi-square tests and I2 statistics.32 The potential for publication bias
a specific condition), description of the system implemented (CPOE/
for each meta-analysis was assessed by inspection of funnel plots
CDSS), and duration of baseline and intervention periods. For stud-
and statistical tests based on weighted linear regression of the inter-
ies that reported mortality, we extracted the number of patient
vention effect on its standard error.33 Subgroup meta-analyses were
deaths in ICU (ICU mortality) and/or the number of ICU patient
also conducted by study quality and ICU type (adult/pediatric) when
deaths in hospital (hospital mortality) at baseline and intervention.
appropriate. Studies conducted in neonatal ICUs34–36 were catego-
For studies that measured LOS, we extracted the mean patient LOS
rized as pediatric. Sensitivity analyses were conducted using differ-
in ICU (ICU LOS) and/or the mean LOS of ICU patients in hospital
ent measurements for the outcomes, such as odds ratios and
(hospital LOS) at baseline and intervention. For studies that re-
standardized mean differences. All statistical tests were 2-sided and
ported medication errors, we extracted the number of prescriptions,
were evaluated at a significance level of 0.05. Analyses were carried
prescribing error definitions, total number of prescribing errors at
out using R version 3.2.1.37
baseline and intervention, and evidence of any new types of errors
introduced by the CPOE system.

Quality assessment RESULTS

We assessed the methodological quality of the included studies using Study characteristics
the Effective Public Health Practice Project (EPHPP) quality assess- Twenty studies met our inclusion criteria: 16 that assessed the tran-
ment tool for quantitative studies.19 The tool was selected because it sition from paper-based ordering to CPOE23,25–30,34–36,38–43 and 4
can be used to assess the methodological quality of both randomized that examined the addition of a targeted CDSS to an existing CPOE
and nonrandomized studies, and has been judged suitable for use in system21,22,44,45 (Table 1). Eleven studies were conducted in the
systematic reviews.20 Using the tool, studies are attributed a rating United States, 4 in the UK, and 1 each in Belgium, Canada, Israel,
of strong, moderate, or weak based on 6 components: (1) selection Saudi Arabia, and Spain. Study publication dates ranged from 2004
bias, (2) study design, (3) confounders, (4) blinding, (5) data collec- to 2014. All studies used a pre-post study design, except 1,23 which
tion methods, and (6) withdrawal and dropouts. was a prospective controlled study. The CPOE vendors included:
Cerner39,40,43–45, GE Centricity21,23,36,42, MetaVision iMDsoft27,30,
Statistical analysis Horizon Expert Orders35,41, Misys QuadraMed26, Global Domin-
We categorized the included studies into those that evaluated the im- ion Access28, IntelliVue Philips29, INVISION Siemens34, and
pact of moving from paper-based ordering to CPOE and those that EPIC.38 Based on the EPHPP quality assessment tool, 13 stud-
evaluated the impact of adding a targeted CDSS to an existing ies22,26–28,34,36,39–45 were rated as being of a moderate methodologi-
CPOE system. We then grouped the studies by outcome measure cal quality and 7 studies21,23,25,29,30,35,38 were rated as weak. No
(medication errors, ICU LOS, hospital LOS, ICU mortality, or hos- studies were rated as strong.
pital mortality). Three studies21–23 reported ICU LOS using median
and interquartile ranges. We contacted the authors of these studies
and requested mean and standard deviation (SD) data. We received Studies comparing CPOE to paper
the results for 1 study;23 however, the authors for the other 2 studies The 16 studies that assessed the transition from paper-based order-
were no longer in possession of the raw data, so these studies could ing to CPOE contained sufficient information to perform
not be included in meta-analysis for this outcome measure. Mean meta-analysis for 4 outcomes: medication errors, ICU LOS, ICU
and SD were estimated24 for 1 study25 that reported ICU LOS using mortality, and hospital mortality. A summary of the findings of
median and range. One study26 reported results on ICU LOS, ICU these 16 studies is provided in Appendix B (available as a Supplemen
mortality, and hospital mortality from 4 overlapping study periods. tary File).
416 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

Figure 2. Relative risk of medication errors (errors indicates number of errors and orders indicates number of orders audited; RE ¼ random effect).

Figure 3. Mean difference of ICU LOS (N indicates number of patients and mean indicates LOS in days; RE ¼ random effect).

Medication errors different methods used to determine error rates. Subsequently, there
Of the 10 studies examining the impact of CPOE on medication er- was significant variation in the frequency of errors, with pre-CPOE
rors, 923,27–30,35,38,41,42 were included in the meta-analysis. The error rates ranging between 4.5%38 and 58.2%,35 and post-CPOE
broad definition of medication prescribing errors was similar across error rates ranging between 0%27 and 8.2%.29 There was evidence
the studies and included illegible, erroneous, or omitted information of heterogeneity between studies (I2 ¼ 99.65%, P < .0001). Seven of
(Appendix C, available as a Supplementary File). However, the level the studies reported a significant reduction in medication errors fol-
of detail regarding the specific elements that were included varied lowing CPOE implementation, 1 study30 reported no change, and 1
between the studies. Some studies, for example, indicated that miss- study38 reported a significant increase in errors. Overall, there was
ing weight41 or no signature29,42 constituted an error of omission evidence that the introduction of CPOE was associated with a signif-
and some included rule violations,30,41 while other studies did not icant reduction in the medication error rate by 85% (pooled RR:
list these elements in their error definitions. There were also 0.15, 95% confidence interval (CI), 0.03–0.80, P ¼ .03) (Figure 2).
Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2 417

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

Figure 4. Relative risk of ICU mortality (event indicates number of deaths and total indicates number of patients; RE ¼ random effect).

Figure 5. Relative risk of hospital mortality (event indicates number of deaths and total indicates number of patients; RE ¼ random effect).

There was no evidence of publication bias (P ¼ .07). Sensitivity anal- weight neonates receiving gentamycin on admission), while the stud-
ysis using results from the post-CPOE–only period (instead of the ies included in the meta-analysis looked at all ICU patients during
post-CPOE and CDSS period) in the Kadmon et al. study30 still indi- their respective study periods. Cordero et al.’s34 study found 14 er-
cated an overall significant error reduction (pooled RR: 0.15, 95% rors in 105 orders examined in the baseline period. In the interven-
CI, 0.03–0.72, P ¼ .02). tion period, no errors were found among the 89 orders examined.
Subgroup analysis by ICU type showed no evidence of significant Of the types of errors reported across the studies, elimination of
error reduction following CPOE implementation for studies conducted illegible orders was the most frequently reported benefit following
in adult ICUs (pooled RR: 0.11, 95% CI, 0.00–3.41, P ¼ .1) or in pedi- CPOE implementation.28,29,35,41 Three studies reported new error
atric ICUs (pooled RR: 0.21, 95% CI, 0.02–2.65, P ¼ .1). Likewise, types arising due to CPOE. Armada et al.28 and Colpaert et al.23
subgroup analysis of 4 studies27,28,41,42 with a quality rating of moder- identified problems with duplicate prescriptions, while Armada
ate showed no reduction in medication errors following CPOE imple- et al.28 and Shulman et al.42 both found problems with erroneous se-
mentation (pooled RR: 0.04, 95% CI, < 0.01–2.82, P ¼ .1). lection from dropdown menus, with Shulman et al.42 indicating that
A study by Cordero et al.34 provided data on medication errors selection of wrong dose from a dropdown menu resulted in 1 poten-
that was not included for synthesis via meta-analysis. The study tially fatal intercepted error. However, Shulman et al.42 also found
looked at errors within a subgroup of ICU patients (very low birth that the frequency of errors considered moderate/major decreased
418 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2

Table 1. Characteristics of the included studies

Author (year) Country ICU type Sample CPOE vendor CPOE/CDSS description Outcomes reported

Studies Al-Dorzi et al. Saudi Arabia Adult All patients Misys, Quad- CPOE with interaction and ICU LOS, Hospital
Comparing (2011)26 raMed allergy alerts, order sets, LOS, ICU Mortal-
CPOE to dose checking, and proto- ity, Hospital
Paper cols Mortality
Ali et al. (2010)27 UK Adult All patients MetaVision, CPOE with default prescrip- Medication Errors
iMDsoft tions
Armada et al. Spain Adult All patients Global CPOE with protocols, and ICU LOS, ICU Mor-
(2014)28 Dominion duplicate, allergy and in- tality, Medication
Access teraction alerts Errors
Carayon and United States Adult All patients EPIC CPOE ICU LOS, Medication
Wood (2009)38 Errors

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

Colpaert et al. Belgium Adult All patients GE Centricity CPOE with protocols, inter- ICU LOS, Medication
(2006)23 action alerts, allergy status, Errors
drug-related complications
Cordero et al. United States Pediatric Very-low- INVISION, CPOE with order sets, ICU Mortality,
(2004)34 birth-weight Siemens allergy, interaction and Medication Errors
infants duplicate alerts, dose
checking and calculation
Del Beccaro et al. United States Pediatric All patients Cerner CPOE with order sets, allergy ICU LOS, ICU
(2006)39 checking, and dose Mortality
checking
Han et al. United States Pediatric Patients ad- Cerner CPOE with allergy and drug Hospital Mortality
(2005)40 mitted via interaction alerts
inter-facility
transport
Jozefczyk et al. United States Pediatric All patients Horizon CPOE Medication Errors
(2013)35 Expert
Orders
Kadmon et al. Israel Pediatric All patients MetaVision, CPOE with dose checking ICU Mortality, Medi-
(2009)30 iMDsoft and default prescriptions cation Errors
Keene et al. UK Pediatric All patients GE Centricity CPOE with order sets Hospital Mortality
(2007)36
Longhurst et al. USA Pediatric All patients Cerner CPOE with clinical decision ICU Mortality
(2010)43 support
Potts et al. United States Pediatric All patients Horizon CPOE with order sets, inter- ICU LOS, Medication
(2004)41 Expert action and allergy alerts, Errors
Orders and dose checking
Shulman et al. UK Adult All patients GE Centricity CPOE with links to drug Medication Errors
(2005)42 information
Thompson et al. Canada Adult Patients with Not reported CPOE with order sets ICU LOS, Hospital
(2004)25 urgent or LOS, Hospital
stat orders Mortality
Warrick et al. UK Pediatric All patients IntelliVue, CPOE with order sets and Medication Errors
(2011)29 Philips drug information
Studies Adams et al. United States Pediatric All patients Cerner Red blood cell transfusion Hospital LOS,
Evaluating (2011)44 alert Hospital Mortality
Targeted Fernandez-Perez United States Adult Patients with GE Centricity Red blood cell transfusion al- ICU LOS, Hospital
CDSS et al. (2007)21 anemia gorithm LOS, ICU Mortal-
ity, Hospital Mor-
tality
Pageler et al. United States Pediatric All patients Cerner Rule that restricts scheduling ICU LOS, Hospital
(2013)45 repeat orders LOS, Hospital
Mortality
Rana et al. United States Adult Patients with Not reported Red blood cell transfusion al- ICU LOS, ICU
(2006)22 anemia gorithm Mortality, Hospital
Mortality

ICU ¼ intensive care unit; CPOE ¼ computerized provider order entry; CDSS ¼ clinical decision support system; LOS ¼ length of stay.
Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2 419

Table 2. LOS and mortality findings in studies evaluating targeted CDSS

Author Baseline Intervention P-value

Duration, months Patients (n) LOS (day) Duration Patients (n) LOS (day)

ICU LOS
Fernandez-Perez et al.21 12 1110 Median 1.7 12 1110 Median 1.7 .18
IQR 0.9–2.9 IQR 0.9–3.5
Pageler et al.45 12 818 Mean 5.1 12 1021 Mean 4.2 .05*
SD 0.7 SD 0.6
Rana et al.22 3 440 Median 1.9 3 403 Median 1.9 .36
IQR 0.9–4.0 IQR 1.0–4.3
Hospital LOS
Adams et al.44 12 809 Mean 16.4 12 1044 Mean 11.5 .0002*

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

SD 31.3 SD 25.8
Fernandez-Perez et al.21 12 1110 Median 9.3 12 1110 Median 9.5 .76
IQR 6.0–17.0 IQR 5.0–17.0
Pageler et al.45 12 818 Mean 16.8 12 1021 Mean 11.6 <.001*
SD 2.1 SD 1.6

Duration Patients (n) Mortality** Duration Patients (n) Mortality** P-value

ICU Mortality
Fernandez-Perez et al.21 12 1110 0.05 12 1110 0.07 .21
Rana et al.22 3 440 0.06 3 403 0.08 .25
Hospital Mortality
Adams et al.44 12 809 0.04 12 1044 0.03 .38
Fernandez-Perez et al.21 12 1110 0.10 12 1110 0.13 .04*
Pageler et al.45 12 818 0.04 12 1021 0.03 .32
Rana et al.22 3 440 0.12 3 403 0.15 .23

*Significant at 0.05 level.

**Mortality provided as rate per patient.
ICU ¼ intensive care unit; LOS ¼ length of stay; CDSS ¼ clinical decision support system; n ¼ number; IQR ¼ interquartile range; SD ¼ standard deviation.

from 1.8% to 0.9% of audited orders. Colpaert et al.23 similarly re- introduction of CPOE reduced ICU mortality by 12% (pooled RR:
ported a decrease in serious medication prescribing errors, from 0.89, 95% CI, 0.78–0.99, P ¼ 0.04; Figure 4). There was no evi-
4.9% to 1.8% of audited orders. dence of publication bias (P ¼ .7). Subgroup analysis of the 4 pediat-
ric studies30,34,39,43 showed no significant change in ICU mortality
ICU LOS (pooled RR: 0.84, 95% CI, 0.60–1.19, P ¼ .2), while subgroup anal-
Seven studies23,25,26,28,38,39,41 were included in the meta-analysis exam- ysis of the 5 studies with a quality rating of moderate26,28,34,39,43 re-
ining the association between CPOE introduction and ICU LOS. There vealed a 14% reduction in ICU mortality after CPOE
was evidence of heterogeneity between studies (I2 ¼ 54.42%, P ¼ .02). implementation (pooled RR: 0.86, 95% CI, 0.78–0.96, P ¼ .02).
Only 1 study reported a significant finding, with ICU LOS found to de-
crease from a mean of 7.44 days to 5.96 days following CPOE imple- Hospital mortality
mentation.23 Overall, there was no evidence of change in ICU LOS Four studies25,26,36,40 were included in the meta-analysis examining the
following the introduction of CPOE (pooled mean difference: 0.10, impact of CPOE on hospital mortality for ICU patients. There was evi-
95% CI, 0.81–0.60, P ¼ .7; Figure 3). There was no evidence of publi- dence of heterogeneity between studies (I2 ¼ 82.53%, P ¼ .0006). Only
cation bias (P ¼ .2). Subgroup analysis of the 5 adult ICU stud- 1 study reported a significant finding, with mortality found to increase
ies23,25,26,28,38 (pooled mean difference: 0.01, 95% CI, 1.16–1.13, from 39 deaths in 790 patients to 36 deaths in 312 patients following
P ¼ .9) and the 4 studies with a quality rating of moderate26,28,39,41 the introduction of CPOE.40 Overall, however, there was no significant
(pooled mean difference: 0.40, 95% CI, 1.07–0.26, P ¼ .1) showed association between CPOE introduction and hospital mortality (pooled
no evidence of reduction in ICU LOS following CPOE implementation. RR: 1.17, 95%CI, 0.53–2.54, P ¼ .6; Figure 5). There was no evidence
Two studies25,26 reported on hospital LOS for ICU patients; of publication bias (P ¼ .5). Subgroup analysis of the 3 studies with a
however, the data were insufficient to perform meta-analysis. Nei- quality rating of moderate26,36,40 revealed similar results to the overall
ther study reported a significant change in hospital LOS following finding (pooled RR: 1.20, 95%CI, 0.28–5.24, P ¼ .6).
the implementation of CPOE.
Studies evaluating targeted CDSSs
ICU mortality Four studies examined the addition of a targeted CDSS to an exist-
Six studies26,28,30,34,39,43 were included in the meta-analysis examin- ing CPOE system21,22,44,45 and reported outcomes on ICU LOS,
ing the association between CPOE introduction and ICU mortality. hospital LOS, ICU mortality, and hospital mortality (Table 2).
The studies were found to be homogenous (heterogeneity between A study that examined the impact of a rule restricting the sched-
studies: I2 ¼ 0%, P ¼ .8). Overall, there was evidence that the uling of repeat orders (i.e., complete blood cell counts, chemistry,
420 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2

49
and coagulation studies within a 24-h interval) in a pediatric setting Han et al., for example, suggested that the negative outcomes
reported a significant decrease in both ICU LOS (from a mean of 5.1 they identified were affected by a combination of order delays due
days to 4.2 days) and hospital LOS (from a mean of 16.8 days to to the inability to “pre-register” patients into the system, the in-
11.6 days).45 Another study in a pediatric setting also reported a sig- creased time required to enter orders at computer terminals located
nificant decrease in hospital LOS (from a mean of 16.4 days to 11.5 away from the patient bedside, the reduction of staff interaction,
days) following the introduction of a red blood cell transfusion algo- and delays in medication administration due to the relocation of
rithm.44 The 2 studies conducted in adult ICUs did not find signifi- drugs from the ward to a centralized pharmacy service. Another
cant changes in ICU LOS21,22 or hospital LOS21 following the unintended consequence of CPOE implementation can be the emer-
addition of a targeted CDSS to an existing CPOE system. gence of new system-related errors.50,51 Among the 10 studies in-
Only 1 study reported a significant change in hospital mortality cluded in this review that examined medication errors, only a few
for ICU patients, from a rate of 0.10 deaths per patient to 0.13 assessed the errors that occurred following the implementation of
deaths per patient, following the addition of a red blood cell transfu- CPOE and identified duplicate prescriptions and erroneous selection
sion algorithm to an existing CPOE system in an adult ICU.21 There from dropdown menus as new system-related errors. These few stud-

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

were no significant findings among the other 3 studies22,44,45 that ies also found that the severity of errors changed, with the frequency
examined hospital mortality, nor the 2 studies21,22 that assessed ICU of serious errors decreasing by > 50%. The limited evidence base of
mortality. the types of errors and potential new risks occurring in critical care
settings post-CPOE implementation underscores the importance of
future research that not only quantifies the changes in error rates and
DISCUSSION patient outcomes but endeavors to understand the nature of these
The transition from paper-based ordering to commercial CPOE sys- changes. Such information is critical to ongoing improvement in the
tems in ICUs was found to be associated with an 85% reduction in design of CPOE systems and the delivery of safe patient care.
medication prescribing error rates. This significant decrease in medi- While we identified a significant overall reduction in medication-
cation errors is consistent with reviews of CPOE implementation in prescribing error rates following CPOE implementation, we also
other inpatient settings.7–11 Overall meta-analysis of LOS and hospi- found that the frequency of medication errors found in each inde-
tal mortality outcomes did not demonstrate a significant change fol- pendent study varied substantially. This was particularly evident in
lowing commercial CPOE implementation in ICU, which is also in baseline error rates, which ranged between 4.5% and 58.2%,
line with other inpatient settings.10,12 However, analysis of ICU whereas post-CPOE error rates ranged between 0% and 8.2%.
mortality showed CPOE implementation to be associated with a Inherent differences between study settings may account for some of
12% mortality risk reduction in ICUs. this variation. However, differences among definitions as to what
In 2005, Han et al.40 reported the findings from a study that in- constitutes a medication prescription error, as well the methods used
cluded 1102 pediatric patients admitted via interfacility transport to detect errors, are the likely cause of the majority of disparity be-
directly to ICU: 790 patients during a 13-month baseline period and tween studies.6,52 Some studies, for example, indicated that missing
312 patients during a 5-month post-CPOE implementation period. weight or no signature constituted an error of omission and some in-
The findings revealed a significant increase in mortality among the cluded rule violations, while other studies did not list these elements
cohort of patients following implementation of a commercial CPOE in their error definitions. These findings reiterate previous calls for
system; from 39 at baseline to 36 post-CPOE (P < .001).40 While the need to use a more standardized set of criteria when defining
such findings are cause for concern, subsequent studies of critically and reporting medication errors. Reckmann et al.,52 for example,
ill patients, both in pediatric30,36,39 and adult26,28 ICUs, have not suggest that future studies should include a clear definition of pre-
demonstrated any significant change in mortality following CPOE scribing errors; absolute error rates pre- and post-CPOE; appropri-
implementation. However, a consistent issue across all of these stud- ate denominators, such as total number of orders; proportions of
ies, including Han et al.,40 is small sample sizes that may not be errors categorized according to a standardized severity scale; and ap-
powered to detect a true effect.46 Conversely, a 2010 study of a propriate significance testing. Such information would facilitate
commercial CPOE by Longhurst et al.43 implemented at a pediatric more accurate comparison between studies.
hospital found a significant decrease in mortality. The study in- The significant reduction in medication error rates following
cluded a substantial sample of 80 063 patient discharges spanning a CPOE implementation is not surprising. The automation and stan-
6-year baseline period and 17 432 patient discharges during the 18- dardization of the format and structure of electronic orders intrinsi-
month post-CPOE implementation period. They found that the cally eliminates some error types, such as legibility errors. While
mean monthly adjusted hospitalwide mortality rate decreased by eliminating these types of errors is important, the greater challenge
20%. Within the current review, the individual studies that assessed is enabling appropriate, evidence-based care. It is here that the im-
ICU mortality did not demonstrate an effect. Increasing the statisti- plementation of comprehensive CPOE applications that include so-
cal power through meta-analysis found a positive effect (even with phisticated CDSSs are anticipated to have the greatest impact on
the use of a conservative approach) of ICU mortality reduction in errors and adverse outcomes.7 However, there is currently very lim-
critically ill populations following CPOE implementation. These ited evidence on the impact of adding targeted CDSSs into existing
findings highlight the importance of future studies that include commercial CPOE systems in ICUs. Among the 4 studies we identi-
larger sample sizes that are sufficiently powered to accurately and fied, the study findings of patient outcomes proved to be mixed.
reliably detect clinically relevant rates of change in important indica- While studies found that the implementation of CDSSs enhanced the
tors following CPOE system introduction. adoption of evidence-based recommendations, this positive impact
Han et al.’s40 study also served to demonstrate the need to moni- on the process of care in ICUs did not necessarily translate into im-
tor outcomes following system implementation, particularly as it is proved patient outcomes. Adams et al.44 and Pageler et al.45 re-
now well recognized that system implementations can result in un- ported significant decreases in LOS following the addition of
anticipated work process changes and unintended consequences.47– CDSSs, while other studies found no change. With regard to
Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2 421

mortality, Fernandez Perez et al.21 reported an increase in unad- SUPPLEMENTARY MATERIAL

justed hospital mortality, but no change in ICU mortality, while the
Supplementary material are available at Journal of the American
other studies found no change in mortality following the addition of
Medical Informatics Association online.
CDSSs. This suggests the need for more sophisticated multilevel sta-
tistical approaches in a much needed area of research, as examina-
tions of mortality and LOS need to account for many complex REFERENCES
variables, including acuity and patient demographics.
1. Institute of Medicine. Preventing Medication Errors. Washington, DC:
The National Academies Press; 2007.
2. Classen DC, Pestotnik SL, Evans R, et al. Adverse drug events in hospital-
Limitations
ized patients: excess length of stay, extra costs, and attributable mortality.
An inherent limitation of systematic reviews is that the soundness of JAMA 1997;277(4):301–6.
the review findings is reliant on the quality of the included studies. 3. Hug BL, Keohane C, Seger DL, et al. The costs of adverse drug events in
We rated the quality of the studies included in this review as either community hospitals. Jt Comm J Qual Patient Saf 2012;38(3):120–6.
methodologically strong, moderate, or weak based on the EPHPP

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

4. Rothschild JM, Landrigan CP, Cronin JW, et al. The critical care safety
quality assessment tool criteria.19 While we found the majority of study: the incidence and nature of adverse events and serious medical er-
studies to be of moderate quality (13 of 20 studies), there were no rors in intensive care. Crit Care Med 2005;33(8):1694–700.
studies rated as strong. As such, in addition to evaluating the find- 5. Cullen DJ, Sweitzer BJ, Bates DW, et al. Preventable adverse drug events
ings from the current evidence base, our study also highlights key in hospitalized patients: a comparative study of intensive care and general
care units. Crit Care Med 1997;25(8):1289–97.
areas where there is a need for more robust studies with larger sam-
6. Moyen E, Camiré E, Stelfox HT. Clinical review: medication errors in crit-
ple sizes in order to ascertain the true effect of the implementation
ical care. Crit Care 2008;12(2):208.
of CPOE systems and CDSSs. While previous systematic reviews on 7. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician or-
the impact of CPOE in other hospital settings have found the evi- der entry and clinical decision support systems on medication safety: a sys-
dence to be largely US-based as a result of including homegrown sys- tematic review. Arch Intern Med 2003;163(12):1409–16.
tems,11 a strength of our review is that we focused on commercial 8. Ammenwerth E, Schnell-Inderst P, Machan C, et al. The effect of elec-
systems. As such, the evidence base we identified was more global, tronic prescribing on medication errors and adverse drug events: a system-
with half of the included studies conducted outside the US, making atic review. J Am Med Inform Assoc 2008;15(5):585–600.
it more applicable to international settings. 9. Nuckols TK, Smith-Spangler C, Morton SC, et al. The effectiveness of
computerized order entry at reducing preventable adverse drug events and
medication errors in hospital settings: a systematic review and meta-analy-
sis. Syst Rev 2014;3:56.
CONCLUSION 10. van Rosse F, Maat B, Rademaker CM, et al. The effect of computerized
Critical care settings, both adult and pediatric, involve unique com- physician order entry on medication prescription errors and clinical out-
come in pediatric and intensive care: a systematic review. Pediatrics
plexities that make them vulnerable to medication errors and ad-
2009;123(4):1184–90.
verse patient outcomes. While limited, the current evidence base
11. Georgiou A, Prgomet M, Paoloni R, et al. The effect of computerized pro-
suggests that the implementation of commercial CPOE systems can vider order entry systems on clinical care and work processes in emergency
significantly decrease the frequency of medication prescribing error departments: a systematic review of the quantitative literature. Ann Emerg
rates, as well as reducing the risk of mortality in ICUs. Future stud- Med 2013;61(6):644–53. e16.
ies that aim to examine medication errors and patient outcomes 12. Thompson G, O’Horo JC, Pickering BW, et al. Impact of the electronic medi-
should ensure they have sufficient sample sizes that are powered to cal record on mortality, length of stay, and cost in the hospital and ICU: a sys-
identify the true effect of CPOE implementation. There is also a crit- tematic review and metaanalysis. Crit Care Med 2015;43(6):1276–82.
ical need to understand the nature of errors arising post-CPOE and 13. Classen DC, Bates DW. Finding the meaning in meaningful use. N Engl J
how the addition of advanced CDSSs can be used to provide even Med 2011;365(9):855–8.
14. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for system-
greater benefit to delivering safe and effective patient care.
atic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol
2009;62(10):1006–12.
15. Wright A, Sittig DF, Ash JS, et al. Clinical decision support capabilities of
AUTHOR CONTRIBUTIONS commercially-available clinical information systems. J Am Med Inform
Assoc 2009;16(5):637–44.
MP led the systematic search, interpretation, and writing. LL led the
16. Levick DL, Stern G, Meyerhoefer CD, et al. Reducing unnecessary testing
meta-analysis. MP, ZN, LL, and JW reviewed articles for inclusion. in a CPOE system through implementation of a targeted CDS interven-
All authors contributed to the study design, and protocol develop- tion. BMC Med Inform Decis Mak 2013;13(1):43.
ment, and participated in the writing and editing of several versions 17. Abstoss KM, Shaw BE, Owens TA, et al. Increasing medication error re-
of this manuscript. porting rates while reducing harm through simultaneous cultural and
system-level interventions in an intensive care unit. BMJ Qual Saf
2011;20(11):914–22.
18. Sintchenko V, Iredell JR, Gilbert GL, et al. Handheld computer-based de-
CONFLICTS OF INTEREST
cision support reduces patient length of stay and antibiotic prescribing in
None to declare. critical care. J Am Med Inform Assoc 2005;12(4):398–402.
19. Thomas H. Quality Assessment Tool for Quantitative Studies. McMaster
University, Toronto: Effective Public Health Practice Project; 2003.
20. Deeks JJ, Dinnes J, D’Amico R, et al. Evaluating non-randomised inter-
FUNDING vention studies. Health Technol Assess 2003;7(27):1–173.
This work was funded by a National Health and Medical Research Council 21. Fernandez Perez ER, Winters JL, Gajic O. The addition of decision
program grant (APP1054146). support into computerized physician order entry reduces red blood cell
422 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. 2

transfusion resource utilization in the intensive care unit. Am J Hematol 38. Carayon P, Wood KE. CPOE Implementation in ICUs. Rockville, MD:
2007;82(7):631–3. The Agency for Healthcare Research and Quality; 2009.
22. Rana R, Afessa B, Keegan MT, et al. Evidence-based red cell transfusion 39. Del Beccaro MA, Jeffries HE, Eisenberg MA, et al. Computerized provider
in the critically ill: quality improvement using computerized physician or- order entry implementation: no association with increased mortality rates
der entry. Crit Care Med 2006;34(7):1892–7. in an intensive care unit. Pediatrics 2006;118(1):290–5.
23. Colpaert K, Claus B, Somers A, et al. Impact of computerized physician 40. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mor-
order entry on medication prescription errors in the intensive care unit: a tality after implementation of a commercially sold computerized physician
controlled cross-sectional trial. Crit Care 2006;10(1):R21. order entry system. Pediatrics 2005;116(6):1506–12.
24. Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from 41. Potts AL, Barr FE, Gregory DF, et al. Computerized physician order en-
the median, range, and the size of a sample. BMC Med Res Methodol try and medication errors in a pediatric critical care unit. Pediatrics
2005;5:13. 2004;113(1 Pt 1):59–63.
25. Thompson W, Dodek PM, Norena M, et al. Computerized physician or- 42. Shulman R, Singer M, Goldstone J, et al. Medication errors: a prospective
der entry of diagnostic tests in an intensive care unit is associated with im- cohort study of hand-written and computerised physician order entry in
proved timeliness of service. Crit Care Med 2004;32(6):1306–9. the intensive care unit. Crit Care 2005;9(5):R516–21.
43. Longhurst CA, Parast L, Sandborg CI, et al. Decrease in hospital-wide

Downloaded from https://academic.oup.com/jamia/article/24/2/413/2907924 by guest on 19 December 2023

26. Al-Dorzi HM, Tamim HM, Cherfan A, et al. Impact of computerized phy-
sician order entry (CPOE) system on the outcome of critically ill adult pa- mortality rate after implementation of a commercially sold computerized
tients: a before-after study. BMC Med Inf Decis Mak 2011;11:71. physician order entry system. Pediatrics 2010;126(1):14–21.
27. Ali J, Barrow L, Vuylsteke A. The impact of computerised physician order 44. Adams ES, Longhurst CA, Pageler N, et al. Computerized physician order
entry on prescribing practices in a cardiothoracic intensive care unit. entry with decision support decreases blood transfusions in children.
Anaesthesia 2010;65(2):119–23. Pediatrics 2011;127(5):e1112–9.
28. Armada ER, Villamanan E, Lopez-de-Sa E, et al. Computerized physician 45. Pageler NM, Franzon D, Longhurst CA, et al. Embedding time-limited
order entry in the cardiac intensive care unit: effects on prescription errors laboratory orders within computerized provider order entry reduces labo-
and workflow conditions. J Crit Care 2014;29(2):188–93. ratory utilization. Pediatr Crit Care Med 2013;14(4):413–9.
29. Warrick C, Naik H, Avis S, et al. A clinical information system reduces 46. Button KS, Ioannidis JPA, Mokrysz C, et al. Power failure: why small sam-
medication errors in paediatric intensive care. Intensive Care Med ple size undermines the reliability of neuroscience. Nat Rev Neurosci
2011;37(4):691–4. 2013;14(5):365–76.
30. Kadmon G, Bron-Harlev E, Nahum E, et al. Computerized order entry 47. Ash JS, Sittig DF, Dykstra RH, et al. Categorizing the unintended socio-
with limited decision support to prevent prescription errors in a PICU. technical consequences of computerized provider order entry. Int J Med
Pediatrics 2009;124(3):935–40. Inform 2007;76:S21–7.
31. Knapp G, Hartung J. Improved tests for a random effects meta-regression 48. Bloomrosen M, Starren J, Lorenzi NM, et al. Anticipating and addressing
with a single covariate. Stat Med 2003;22(17):2639–710. the unintended consequences of health IT and policy: a report from the
32. Higgins JPT, Thompson SG, Deeks JJ, et al. Measuring inconsistency in AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc
meta-analyses. BMJ 2003;327(7414):557–60. 2011;18(1):82–90.
33. Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis de- 49. Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of informa-
tected by a simple, graphical test. BMJ 1997;315(7109):629–34. tion technologies in health care - an interactive sociotechnical analysis.
34. Cordero L, Kuehn L, Kumar RR, et al. Impact of computerized physician J Am Med Inform Assoc 2007;14(5):542–9.
order entry on clinical practice in a newborn intensive care unit. J Perina- 50. Westbrook JI, Baysari MT, Li L, et al. The safety of electronic prescribing:
tol 2004;24(2):88–93. manifestations, mechanisms, and rates of system-related errors associated
35. Jozefczyk KG, Kennedy WK, Lin MJ, et al. Computerized prescriber order with two commercial systems in hospitals. J Am Med Inform Assoc
entry and opportunities for medication errors: comparison to tradition 2013;20(6):1159–67.
paper-based order entry. J Pharm Pract 2013;26(4):434–7. 51. Westbrook JI, Reckmann M, Li L, et al. Effects of two commercial elec-
36. Keene A, Ashton L, Shure D, et al. Mortality before and after initiation of tronic prescribing systems on prescribing error rates in hospital in-
a computerized physician order entry system in a critically ill pediatric patients: a before and after study. PLoS Med 2012;9(1):e1001164.
population. Pediatr Crit Care Med 2007;8(3):268–71. 52. Reckmann MH, Westbrook JI, Koh Y, et al. Does computerized provider
37. R Development Core Team. R: A Language and Environment for Statistical order entry reduce prescribing errors for hospital inpatients? A systematic
Computing. Vienna, Austria: R Foundation for Statistical Computing; 2013. review. J Am Med Inform Assoc 2009;16(5):613–23.