DOI: 10.1111/eci.

12834

PERSPECTIVE

How to survive the medical misinformation mess

John P. A. Ioannidis*,†,‡, Michael E. Stuart§,¶, Shannon Brownlee**,†† and Sheri A. Strite¶
*
Departments of Medicine, Health Research and Policy, and Biomedical Data Science, Stanford University School of Medi-
cine, Stanford, CA, USA, †Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA,
‡
Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA, §Department of
Family Medicine, University of Washington School of Medicine, Seattle, WA, USA, ¶Delfini Group LLC, Seattle, WA, USA,
**
Lown Institute, Brookline, MA, USA, ††Department of Health Policy, Harvard T.H. Chan School of Public Health, Cambridge,
MA, USA

ABSTRACT
Most physicians and other healthcare professionals are unaware of the pervasiveness of poor quality clinical
evidence that contributes considerably to overuse, underuse, avoidable adverse events, missed opportunities
for right care and wasted healthcare resources. The Medical Misinformation Mess comprises four key pro-
blems. First, much published medical research is not reliable or is of uncertain reliability, offers no benefit to
patients, or is not useful to decision makers. Second, most healthcare professionals are not aware of this
problem. Third, they also lack the skills necessary to evaluate the reliability and usefulness of medical evidence.
Finally, patients and families frequently lack relevant, accurate medical evidence and skilled guidance at the time
of medical decision-making. Increasing the reliability of available, published evidence may not be an imminently
reachable goal. Therefore, efforts should focus on making healthcare professionals, more sensitive to the lim-
itations of the evidence, training them to do critical appraisal, and enhancing their communication skills so that
they can effectively summarize and discuss medical evidence with patients to improve decision-making. Similar
efforts may need to target also patients, journalists, policy makers, the lay public and other healthcare stake-
holders.

Currently, there are nearly approximately 17 million articles in to meet patients’ needs [11–13]. While there are many causes for
PubMed tagged with ‘human(s)’, with >700 000 articles iden- inappropriate care and waste, much of it may be attributed to
tified as ‘clinical trials’, and >1
8 million as ‘reviews’ (approxi- the poor quality of information that clinicians and patients rely
mately 160 000 as ‘systematic reviews’). Nearly one million on to make decisions about the services they deliver or receive.
articles on humans are added each year [1]. Popular media also We use the term ‘Medical Misinformation Mess’ to encom-
abound with medical stories and advice for patients. pass the set of issues that relate to the low quality of medical
Unfortunately, much of this information is unreliable or of information deeply embedded in clinical processes and deci-
uncertain reliability. Most clinical trials results may be mis- sions. Although the Medical Misinformation Mess affects
leading or not useful for patients [2,3]. Most guidelines (which multiple stakeholders – clinicians, patients, researchers, medi-
many clinicians rely on to guide treatment decisions) do not cal information content developers (e.g. producers of guideli-
fully acknowledge the poor quality of the data on which they nes and decision aids), health journalists, professional
are based [4]. Most medical stories in mass media do not meet associations, policymakers, politicians, hospitals, insurers, drug
criteria for accuracy [5], and many stories exaggerate benefit companies, healthcare advocates and others – here, our focus is
and minimise harms. mainly on clinician and patient issues, and on remedies for
Clinicians and patients often do not recognise how pervasive those aspects.
this problem is and how profoundly it affects the care they The Medical Misinformation Mess comprises four key
deliver or receive. Twenty to 50 per cent of all healthcare ser- problems:
vices delivered in the United States is inappropriate, wasting
resources and/or harming patients [6–10]. Much of this waste is 1 Much published medical research is not reliable or is of
due to overuse of medical interventions, resulting in an uncertain reliability, offers no benefit to patients, or is not
unknown amount of preventable harms. Underuse of effective useful to decision makers.
and safe interventions further compounds the system’s failure 2 Most healthcare professionals are not aware of this problem.

European Journal of Clinical Investigation 1

 J. P. A. IOANNIDIS ET AL. www.ejci-online.com

3 Even if they are aware of this problem, most healthcare

professionals lack the skills necessary to evaluate the relia- Problem 2. Most healthcare professionals are
bility and usefulness of medical evidence. not aware of this problem
4 Patients and families frequently lack relevant, accurate Based on training thousands of attendees at our educational
medical evidence and skilled guidance at the time of medical programmes and professional interactions with colleagues at
decision-making. all levels – from young trainees to top clinical and academic
leadership – we are convinced that very few healthcare pro-
Problem 1. Much published medical research is fessionals are aware of the pervasiveness of biased and inac-
not reliable or is of uncertain reliability, offers no curate medical literature. It is our combined experience that
benefit to patients or is not useful to decision ignorance of this problem, even at the highest levels of aca-
makers demic and clinical leadership is profound.
Evidence for this ignorance emerges also in several studies
With the ever-increasing number of publications, there is a and surveys. In a study of journal reading habits, internists
growing need for well-designed and conducted systematic (approximately half of whom were alumni of the Robert Wood
reviews and meta-analyses to provide valid, cumulative evi- Johnson Clinical Scholars Program) reported they obtained
dence on relevant topics. This need is not easy to meet. information mostly from abstracts and not the full articles,
Although systematic reviews and secondary sources may stating that they relied on editors to assure rigour and study
accelerate evidence uptake [14], most systematic reviews and quality [20]. Such trust may be misplaced. For example, a recent
meta-analyses appear to be either not useful or of uncertain study showed that several editors of peer-reviewed journals
utility [3]. The majority are unnecessary (duplicative), inaccu- could not tell whether a trial was randomised without a special
rate or misleading due to biases in the methodology and checklist. Even then, of the 324 studies editorial staff considered
selective reporting of results, or they address questions that as randomised trials, 127 (39%) were actually not randomised
have no clinical value. [21].
Underlying concerns about the methodology and bias of Many healthcare professionals put too much trust in
systematic reviews and meta-analyses is the quality of the abstracts for filtering the literature, or expect that systematic
published medical research on which they are based [2,3]. reviews or guidelines will get rid of the unreliability and
Usefulness of clinical research [15] requires existence of a nonutility problem. Clearly, the reliability of a study’s results
real problem to address, proper context placement, suffi- cannot be accomplished solely by reading the abstract. One
cient information gain, patient-centeredness, pragmatism, study found that nearly half of abstracts of randomised con-
reasonable value-for-money, nonfutility and transparency. trolled trials contained biased reporting of study results,
Very few clinical studies meet at least six of the eight cri- implying benefit when there was no statistically significant
teria [15]. In one survey of 60 352 studies, a meagre 7% difference in the primary endpoint between study arms [22].
passed criteria of high-quality methods and clinical rele- Flawed primary studies are compounded by flawed systematic
vancy [16] and fewer than 5% passed a validity screening reviews and lead to flawed clinical guidelines that make for
for an evidence-based journal [17]. Uncertain or poor conflicting recommendations unsupported by reliable evidence
quality evidence leaves clinicians, often under pressure, [23,24]. Most healthcare professionals are not even minimally
without definitive information regarding possible treat- aware of these issues.
ments. For brevity, our focus here is on therapeutic inter-
ventions, but similar problems are found in publications Problem 3. Most healthcare professionals lack
dealing with diagnosis, prevention and other areas of skills in being able to evaluate the reliability and
medical care. usefulness of evidence
Many solutions have been proposed for making medical
research more reliable [18,19] and more useful [15], and In our encounters with students, clinicians and others
these will not be discussed in detail here as they have been working in the healthcare industry (including academicians,
covered elsewhere [15,18,19], but this problem is unlikely to researchers, editors, peer reviewers, pharmacists, regulators,
be fixed imminently. Moreover, the large amount of accu- politicians and employees of insurance companies, hospitals,
mulated misleading information is difficult to extract from the pharmaceutical industry and new technology compa-
the literature. Meanwhile, we need to work on the other nies), we have found a lack of the basic skills required for
three components of the misinformation mess to prevent determining a study’s reliability and applicability. For
misleading evidence from flowing downstream into clinical example, in a pretest administered to a sampling of more
decisions. than 500 physicians, clinical pharmacists and other

2 ª 2017 Stichting European Society for Clinical Investigation Journal Foundation

 THE MEDICAL MISINFORMATION MESS

healthcare professionals attending evidence-based medicine care and are neither motivated nor prepared to apply EBM
(EBM) training programmes in 2002 and 2003, 70 per cent skills. Upon entry to residency programmes, their ability to
failed a simple three-question critical appraisal training appraise the medical literature critically is extremely limited
programme test. The three pretest questions were designed [28].
to determine if attendees could recognise the absence of a Currently, strong evidence regarding the most effective
control group, understand the issue of overestimating benefit training approach to equip healthcare professionals with the
when provided with relative risk reduction information required knowledge and skills to consistently apply valid
without absolute difference information and determine research evidence in their daily work is lacking. Studies of
whether an intention-to-treat analysis was performed. Sur- the effectiveness of teaching EBM and critical appraisal of
prisingly, among those who reported feeling confident to medical evidence are heterogeneous in study designs, pop-
evaluate the medical literature, 72 per cent failed the test, ulations, intervention components, outcome measures, study
even with generous criteria for correct answers [25]. We settings, duration and other factors. Several systematic
have repeated the same pretest with various groups each reviews have reported that teaching EBM is effective, but
year with similar results. A well-designed and conducted study details and methodological quality vary widely [29–
trial reported similar findings: clinicians without formal EBM 31]. An overview of reviews [29] found 16 systematic
training score poorly on the 15-test question Berlin Ques- reviews that have tried to cover this topic and more reviews
tionnaire (mean score, 4
2 correct answers compared with were published since then [31]. Most systematic reviews
EBM experts’ mean score of 11
9) [26]. have concluded in favour of the effectiveness of EBM
Critical appraisal skills matter greatly for assuring optimal teaching, but outcomes vary and focus mostly on knowledge
patient care. When practicing clinicians cannot distinguish and skills rather than practical applications, while ran-
between valid and false results, they are at risk of delivering domised trials are relatively few. For example, a Cochrane
useless treatments, or worse, harming their patients. For review of EBM teaching effectiveness [32] concluded that
example, evidence of a fourfold increased risk of myocardial EBM teaching does have positive impact on the knowledge
infarction in patients receiving rofecoxib (Vioxx, Merck, and skills of physicians. This is based on only three RCTs
Whitehouse Station, NJ, USA) as compared to naproxen [33–35] (with total sample size n = 270, shown along with
(Novopharm Biotech, Toronto, Canada) was plainly available risk of bias assessments [36] in Table 1) meeting the inves-
in the abstract of the VIGOR trial. However, peer reviewers, tigators’ criteria after reviewing a total of 11 057 titles and
editors and readers of the New England Journal of Medicine abstracts yielding 148 potentially relevant studies. Another
accepted the spurious argument that naproxen was cardiopro- systematic review [31] of teaching EBM in healthcare pro-
tective. The VIGOR investigators concluded that the increased fessionals excluding physicians and medical students found
risk of myocardial infarction with rofecoxib did not exist, stat- only 13 eligible studies with a total of 1120 participants and
ing without any supporting evidence that the ‘. . .results are of those only four (with 168 participants) were randomised.
consistent with the theory that naproxen has a coronary pro- The durability of the effects and the optimal ways of
tective effect’. Millions of prescriptions were written before the maintaining acquired knowledge and skills are even less
drug was withdrawn from the market in 2004, after several studied.
studies reported significantly increased risks of cardiovascular
events and death [27]. Problem 4. Patients and families frequently lack
The potential risks of delivering poor care might be mitigated relevant, accurate medical evidence and skilled
if healthcare professionals followed trustworthy clinical guidance at the time of medical decision-making
guidelines or based their actions on reliable systematic reviews
People are bombarded with medical news stories, television
and meta-analyses, which ought to weed out false results.
and radio talk shows, social media, pop culture magazines,
However, lack of critical appraisal skills on the part of
spurious websites, direct-to-consumer drug and medical device
reviewers and guideline creators routinely leads to flawed
ads, hospital marketing messages and other media sources,
systematic reviews and guidelines, leaving clinicians with few
much of which are incomplete or wildly inaccurate [37]. Some
resources for sorting fact from fiction [3].
television shows hosted by physicians amount to hucksterism.
The teaching of appraisal skills in medical and other
Today, more media articles have begun to note problems in
schools and other training programmes, such as residencies,
medical science: instances of biased medical research, a lack of
appears at first glance to be fertile ground for providing
evidence for both alternative and allopathic treatments and the
clinicians with needed skills. However, studies assessing
problem of conflict of interest. But many health care and
medical student competencies suggest they frequently do
medical journalists appear to remain largely unaware of the
not see or are not taught the relevance of EBM to clinical

European Journal of Clinical Investigation 3

 J. P. A. IOANNIDIS ET AL. www.ejci-online.com

Table 1 Randomised trials assessing the effectiveness of teaching evidence-based medicine or critical appraisal of medical
evidence to physicians [32]
Study design/size/
Reference population Intervention Outcomes Effect size Risk of bias*

Linzer 44 internal medicine General medicine journal Per cent improvement 26% improvement in Unclear risk of bias
et al. [33] interns at Duke club that emphasised in knowledge using a the intervention Small trial lacking in
University who epidemiologic methods test instrument group compared details of
volunteered and critical appraisal of developed by the with 6% randomisation and
medical evidence; five Delphi method. improvement in the concealment of
journal club sessions control group allocation; minimal
(mean); conducted over (P = 0
02). loss to follow-up;
average of 9
5 months assessors were
led by general medicine blinded.
faculty; control group
received seminars
dealing with ambulatory
medicine issues.
MacRae 81 members of the Internet curriculum in Primary outcome Intervention group Unclear risk of bias
et al. [34] Canadian critical appraisal skills; measure: locally score on Lacking in details of
Association of included a clinical and developed 51 item examination: 58
8% randomisation and
General Surgeons methodologic article, a test to assess validity vs. control group concealment of
who volunteered for listserve discussion of assessment and score of 50% allocation; attrition
6-month Internet- methodology; applicability skills. (P < 0
001). unbalanced and
based study; methodologic critiques; > 20%; adequate
included surgeons 16 articles assessed with blinding of
from most provinces. critical appraisal guide; assessors.
control group received
articles to read and had
access to online critical
appraisal articles.
Taylor 145 self-selected Half-day skills training Knowledge: validated Knowledge score: Unclear risk of bias
et al. [35] general practitioners, based on the Critical tool – 18 multiple- mean difference 2
6 Computer generated
hospital physicians, Appraisal Skills choice questions (95% CI: 0
6–4
6). randomisation
allied health Programme (CASP) focused on Skills assessment: codes; unclear
professionals, developed from knowledge of mean difference: 1
2 concealment of
healthcare educational methods of principles for (95% CI: 0
01–2
4). allocation; balanced
managers/ McMaster University; appraising evidence. groups; attrition
administrators from control group: waiting Skills assessment: incompletely
the south-west of list for workshop. appraisal of a reported; adequate
England. systematic review. blinding of
assessors.
*Risk of bias ratings based on Cochrane Collaboration’s ‘Risk of bias’ assessment tool [36] that examines the following six criteria: sequence generation,
allocation concealment, blinding of participants, personnel and outcome assessors, incomplete outcome data, selective outcome reporting and other sources of
bias.

degree to which the ‘information’ they gather for stories has Informed or misinformed, patients eventually are at the core
been shaped by the interests of manufacturers and research of making medical decisions [41]. The legal doctrine of
universities. Mass media consumers have few means of deter- informed consent requires that patients understand that they
mining the accuracy of any given news item and thus often have treatment choices and the potential benefits and harms of
view evidence through the lens of the mass media. We need to each choice [42], while medical ethics recognises that their
educate the public how to deal with these sources of misin- values and preferences must be honoured [41]. Shared deci-
formation [38–40]. sion-making (SDM) involves clinicians sharing medical

4 ª 2017 Stichting European Society for Clinical Investigation Journal Foundation

 THE MEDICAL MISINFORMATION MESS

evidence with patients, eliciting their values and preferences uncertainties (5%) and rarely (2%) assessed the patient’s under-
and deciding with their patients the best course of treatment. standing of the decision [51]. In a similar study, only rarely (1
1–
Ensuring that patients are adequately prepared to make deci- 16
6%) did physicians relate to patients the uncertainty of evi-
sions usually requires professional assistance to explore both dence surrounding the recommended treatments [52].
the treatment options and the medical evidence, so that the The combination of unreliable medical evidence, the tsu-
potential outcomes that matter most to the patient can be nami of misleading reports in the media, inadequate discus-
accurately determined. This process depends on sufficient, sions between clinicians and patients and a culture of
relevant and valid information, and clarifying discussion that patients’ trust in providers’ recommendations and expectation
confers ‘agency’ – the capacity of an individual to make free of something to be done together produce massive medical
choices [43,44]. misinformation, with suboptimal, nonpatient-centred deci-
Use of SDM and decision support materials (often called sion-making.
patient decision aids) improves decision-making around
many different ‘preference-sensitive’ clinical choices. A sys-
Moving forward
tematic review of 115 randomised controlled trials involving
more than 34 000 patients of the effects of SDM and exposure We think that all healthcare professionals involved in medical
to decision aids (written, electronic, audiovisual or in web- decision-making should possess basic critical appraisal skills
based tool formats), reported that patients had greater and be knowledgeable about which sources of information are
knowledge gain, felt more confident regarding what mattered likely to be accurate and relevant. As Glasziou et al. [53] have
to them and had more accurate expectations about risks and stated, ‘a 21st century clinician who cannot critically read a
benefits than patients who received usual care. Participants in study is as unprepared as one who cannot take a blood pres-
the experimental arms participated more actively in the sure or examine the cardiovascular system’. Such illiteracy is
decision-making process and were on average ~20% more common and clinicians thus foster unrealistic expectations
likely to make conservative choices when facing difficult about medicine. A systematic review of 48 studies on clinician
decisions regarding surgical and nonsurgical interventions, expectations on the benefits and/or harms of treatments, tests
resulting in no known adverse health outcomes, decreased or screening tests showed that in most studies most physicians
satisfaction or anxiety [45]. had inaccurate expectations. Moreover, it was far more com-
Given the power of patient decision aids and clinician–pa- mon for clinicians to overestimate than underestimate benefits
tient dialogue, both need to be accurate if patients are to make and to underestimate rather than overestimate harms [54].
properly informed decisions. Accuracy of decision aids Their inability to assess evidence further contributes to skewed
depends upon the critical appraisal skills of their producers views among patients, the media, policy makers and others.
[46], while the effectiveness of clinician–patient conversations The problem of having so much unreliable and nonuseful
requires clinicians who are willing and able to engage and published medical research may be attacked at its root, that is by
know the evidence. Barriers to implementing effective SDM funding, conducting, publishing and disseminating more true
include pervasive professional indifference, organisational and useful research. However, it is important in the meantime to
inertia, lack of physician comfort with decision aids, time con- make healthcare professionals, patients, journalists and others
straints, competing priorities, lack of training, lack of reim- aware of the problem, provide them with critical appraisal skills
bursement and perceived work burden and cost [47]. Patients’ and ensure that the best evidence available is included in clini-
preferences for a treatment often differ from those of clinicians cian–patient discussions about treatment choices.
[48,49], yet clinicians often underestimate patients’ desires for How to accomplish those three goals is neither obvious nor
information [50]. simple. We need additional high-quality RCTs on the effec-
Not surprisingly, discussions with patients infrequently fulfil tiveness of specific interventions to teach EBM. Assessed
the criteria considered integral to informed decision-making interventions may include both fixed components (e.g. basic
and informed consent. A study of outpatient visits in primary EBM concepts and skills) and variable components (e.g. con-
care clinics assessed six elements of informed decision-making: textual elements such as settings, leadership support, involve-
description of the nature of the decision, discussion of alter- ment of opinion leaders and other details regarding employed
natives; discussion of risks and benefits, discussion of related implementation strategies) [55]. It has not been decisively
uncertainties; assessment of the patient’s understanding and shown which implementation strategies are optimal for a given
elicitation of the patient’s preference. No discussions fulfilled all clinical practice change. Important barriers and considerations
criteria. Physicians frequently described the nature of the deci- for successful clinical practice change may include personal
sion (83%), but infrequently elicited patients’ preferences (19%), factors (e.g. motivation, time, skills required to evaluate the
discussed alternatives (14%), risks and benefits (9%), relevance and validity of medical information),

European Journal of Clinical Investigation 5

 J. P. A. IOANNIDIS ET AL. www.ejci-online.com

recommendation-related factors (access, complexity) and manuscripts to provide all information required for their criti-
external factors (e.g. local clinical culture) [56]. cal appraisal. Government agencies and professional groups
We should also caution that any of the EBM critical appraisal may also be influential stakeholders in ensuring that investi-
tools can be subverted. For example, industry-sponsored trials gators possess key EBM skills. The press also needs training in
may be performed and presented in a way that they tick all the critical thinking [58]. Schools of journalism should include basic
boxes in the CONSORT checklist and on risk of bias tools, even epidemiology and statistics in their coursework for future
as some fundamental aspects of their design, for example the healthcare and medical writers. Journalists and editors should
question asked and how it is asked and answered (what com- also be aware of the evidence-based critiques of mass media
parators, outcomes or follow-up are used), may still be highly stories, such as those offered by HealthNewsReview.org [38].
misleading. There is no standard package or automated train- Schools for healthcare professionals could do a better job of
ing tool to substitute for thinking and some healthy scepticism. ensuring that training in critical appraisal of the medical liter-
Similarly, while decision tools can enhance SDM, automated ature is integrated into the curricula and clinical care. Encour-
tools alone cannot address some additional fundamental chal- aging reports suggest that attitudes, knowledge and critical
lenges that weaken the position and involvement of patients in appraisal skills can improve through tightly integrated EBM
the decision-making. For example, patients typically have had teaching programmes [59,60]. Payers and accreditation bodies,
little or no input to the design of the research that produced the such as the Accreditation Council for Graduate Medical Edu-
available evidence, power imbalances may exist in the clinical cation, involved in the delivery of healthcare, could also require
consultation, and many people do not seek or cannot access skills in critical appraisal of medical evidence.
care [57]. In addition, journalists must be trained to bring Eventually, successful initiatives should be part of everyday
greater scepticism and some critical appraisal skills to reporting clinical experience, not seen as an artificial formal imposed
on medical research. Addressing these challenges requires requirement. Teachers and trainers need ever sharper skills in
rethinking medical research and care at large. critical appraisal of the medical literature [59]. Furthermore, all
Acknowledging these broader challenges, agents of change healthcare professionals can take up the responsibility to mas-
could include journals, government agencies, professional ter skills and become teachers and trainers for themselves
groups, schools for healthcare professionals, payers, accredita- and for others during encounters with patients and decision-
tion bodies, as well as fellow healthcare providers who can making.
reinforce the importance of mastering critical appraisal and
communication skills in every day’s practice. The mass media Acknowledgement
have a special role to play in this regard, as all players in METRICS is funded by a grant from the Laura and John Arnold
healthcare, from journals to clinicians to government agencies, Foundation. The work of John Ioannidis is supported by an
may respond to criticism in the press. unrestricted gift from Sue and Bob O’Donnell.
Critical appraisal skills may have a short half-life and need
continuous use and reinforcement. Moreover, given the vast Address
and rapidly expanding nature of the literature and the limited Departments of Medicine, of Health Research and Policy,
time available to healthcare professionals, it may be easier to and of Biomedical Data Science, Stanford University School
focus on using critically pre-appraised evidence, for example of Medicine, Stanford, CA 94305, USA (J. P. A. Ioannidis);
from well-done evidence synthesis efforts or guidelines, rather Meta-Research Innovation Center at Stanford (METRICS),
than try to appraise every single article. However, even sys- Stanford University, Stanford, CA 94305, USA (J. P. A. Ioanni-
tematic reviews, meta-analyses and guidelines are currently so dis); Department of Statistics, Stanford University School of
numerous and often so poor, biased, conflicted or useless that Humanities and Sciences, Stanford, CA 94305, USA (J. P. A.
building and maintaining skills to appraise them is not an easy Ioannidis); Department of Family Medicine, University of
task. Moreover, becoming proficient in dissecting the caveats of Washington School of Medicine, Seattle, WA 98115, USA (M. E.
higher-level syntheses requires understanding the problems of Stuart); Delfini Group LLC, Seattle, WA 98229, USA (M. E.
primary studies. Stuart, S. A. Strite); Lown Institute, Brookline, MA 02446, USA
As more journal editors recognise the Medical Misinforma- (S. Brownlee); Department of Health Policy, Harvard T.H. Chan
tion Mess as an issue, they can promote awareness by pub- School of Public Health, Cambridge, MA 02115, USA (S.
lishing articles, commentaries and editorials on the subject. It Brownlee).
seems astonishing that there is a need to point out that inves-
tigators should understand what constitutes good design, Correspondence to: John P. A. Ioannidis, Department of Med-
methodology, execution, performance and reporting in icine, Meta-Research Innovation Center at Stanford and Stan-
research; nevertheless, the need exists. Journals should require ford Prevention Research Center, 1265 Welch Rd, MSOB X306,

6 ª 2017 Stichting European Society for Clinical Investigation Journal Foundation

 THE MEDICAL MISINFORMATION MESS

Stanford CA 94305, USA. Tel.: +1 650 7045584; e-mail: jioan- 18 Ioannidis JP. How to make more published research true. PLoS Med
[email protected] 2014;11:e1001747. https://doi.org/10.1371/journal.pmed.1001747.
19 Munaf o MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, du
Sert NP et al. A manifesto for reproducible science. Nat Hum Behav
Received 1 September 2017; accepted 1 September 2017 2017;1:0021. https://doi.org/10.1038/s41562-016-0021.
20 Saint S, Christakis DA, Saha S, Elmore JG, Welsh DE, Baker P et al.
References Journal reading habits of internists. J Gen Intern Med 2000;15:881–4.
1 PubMed search terms: “humans;” “humans” with Clinical Trial 21 Hopewell S, Boutron I, Altman DG, Barbour G, Moher D, Montori V
article type; “humans” with Review article type; “humans” with et al. Impact of a web-based tool (WebCONSORT) to improve the
Systematic Reviews article type. PubMed database. Available at: reporting of randomised trials: results of a randomised controlled
https://www.ncbi.nlm.nih.gov/pubmed. Accessed on 13 February trial. BMC Med 2016;14:199.
2017. 22 Vera-Badillo FE, Napoleone M, Krzyzanowska MK, Alibhai SM,
2 Ioannidis JP. Why most published research findings are false. PLoS Chan AW, Ocana A et al. Bias in reporting of randomized clinical
Med 2005;2:e124. trials in oncology. Eur J Cancer 2016;61:29–35. https://doi.org/10.
3 Ioannidis JP. The mass production of redundant, misleading, and 1016/j.ejca.2016.03.066.
conflicted systematic reviews and meta-analyses. Milbank Q 23 Shaneyfelt T. In guidelines we cannot trust. Arch Intern Med
2016;94:485–514. https://doi.org/10.1111/1468-0009.12210. 2012;172:1633–4.
4 Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP; Guideline 24 Kung J, Miller RR, Mackowiak PA. Failure of clinical practice
Panel Review Working Group. Ensuring the integrity of clinical guidelines to meet Institute of Medicine standards: two more
practice guidelines: a tool for protecting patients. BMJ. 2013;347:f5535. decades of little, if any, progress. Arch Intern Med 2012;172:1628–33.
5 Schwitzer G. A guide to reading health care news stories. JAMA 25 Program pre-test scores: “Using the Medical Literature”. June 6,
Intern Med 2014;174:1183–6. 2003. Available at: http://www.delfini.org/Delfini_Pre-Test_Report_
6 Chassin MR, Galvin RW; the National Roundtable on Health Care 0306.pdf. Accessed on 1 February 2017.
Quality. The urgent need to improve quality: Institute of Medicine 26 Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do
National Roundtable on Health Care Quality [consensus statement]. short courses in evidence based medicine improve knowledge and
JAMA 1998;280:1000–5. skills? Validation of Berlin questionnaire and before and after study
7 Kerr EA, McGlynn EA, Adams J, Keesey J, Asch SM. Profiling the of courses in evidence based medicine. BMJ 2002;7:1338–41.
quality of care in twelve communities: results from the CQI study. 27 James MJ, Cook-Johnson RJ, Cleland LG. Selective COX-2 inhibitors,
Health Aff (Millwood) 2004;23:247–56. eicosanoid synthesis and clinical outcomes: a case study of system
8 McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A failure. Lipids 2007;42:779–85.
et al. The quality of health care delivered to adults in the United 28 Smith AB, Semler L, Rehman EA, Haddad ZG, Ahmadzadeh KL,
States. N Engl J Med 2003;348:2635–45. Crellin SJ et al. A cross-sectional study of medical student knowledge
9 Skinner J, Fisher ES, WennbergJE. For the National Bureau of Economic of evidence-based medicine as measured by the Fresno Test. J Emerg
Research. The efficiency of Medicare. Working Paper No. 8395. Med 2016;. https://doi.org/10.1016/jemermed.2016.02.006.
Cambridge, MA: National Bureau of Economic Research; July 2001. 29 Young T, Rohwer A, Volmink J, Clarke M. What are the effects of
10 Olsen LA, Saunders RS, Yong PL, eds. The Healthcare Imperative: teaching evidence-based health care (EBHC)? Overview of
Lowering Costs and Improving Outcomes: Workshop Series Summary. systematic reviews. PLoS ONE 2014;9:e86706. https://doi.org/10.
Washington, DC: National Academies Press; 2010. 1371/journal.pone.0086706.
11 Brownlee S, Chalkidou K, Doust J, Elshaug AG, Glasziou P, Heath I 30 Flores-Mateo G, Argimon JM. Evidence based practice in
et al. Evidence for overuse of medical services around the world. postgraduate healthcare education: a systematic review. BMC Health
Lancet 2017;390:156–68. Serv Res 2007;7:119.
12 Saini V, Brownlee S, Elshaug AG, Glasziou P, Heath I. Addressing 31 Hecht L, Buhse S, Meyer G. Effectiveness of training in evidence-
overuse and underuse around the world. Lancet 2017;390:105–7. based medicine skills for healthcare professionals: a systematic
13 Glasziou P, Straus S, Brownlee S, Trevena L, Dans L, Guyatt G et al. review. BMC Med Educ 2016;16:103. https://doi.org/10.1186/s12909-
Evidence for underuse of effective medical services around the 016-0616-2.
world. Lancet 2017;390:169–77. pii: S0140-6736(16)30946-1. https://d 32 Horsley T, Hyde C, Santesso N, Parkes J, Milne R, Stewart R.
oi.org/10.1016/s0140-6736(16)30946-1 Teaching critical appraisal skills in healthcare settings. Cochrane
14 Guyatt GH, Meade MO, Jaeschke RZ, Cook DJ, Haynes RB. Database Syst Rev 2011;(11):CD001270. https://doi.org/10.1002/
Practitioners of evidence based care. Not all clinicians need to 14651858.cd001270.pub2.
appraise evidence from scratch but all need some skills. BMJ 33 Linzer M, Brown JT, Frazier LM, DeLong ER, Siegel WC. Impact of a
2000;320:954–5. medical journal club on house-staff reading habits, knowledge, and
15 Ioannidis JP. Why most clinical research is not useful. PLoS Med critical appraisal skills. A randomized control trial. JAMA
2016;13:e1002049. https://doi.org/10.1371/journal.pmed.1002049. 1988;260:2537–41.
16 McKibbon KA, Wilczynski NL, Haynes RB. What do evidence-based 34 MacRae HM, Regehr G, McKenzie M, Henteleff H, Taylor M,
secondary journals tell us about the publication of clinically Barkun J et al. Teaching practicing surgeons critical appraisal skills
important articles in primary healthcare journals? BMC Med with an Internet-based journal club: a randomized, controlled trial.
2004;2:33. Surgery 2004;136:641–6.
17 Glasziou P. The EBM journal selection process: how to find the 1 in 35 Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Critical appraisal skills
400 valid and highly relevant new research articles. Evid Based Med training for health care professionals: a randomized controlled trial.
2006;11:101. BMC Med Educ 2004;4:30.

European Journal of Clinical Investigation 7

 J. P. A. IOANNIDIS ET AL. www.ejci-online.com

36 Higgins JPT, Altman DG (editors). Chapter 8: assessing risk of bias patient uses: shared decision making at Massachusetts General
in included studies. In: Higgins JPT, Green S, editors. Cochrane Hospital. Health Aff (Millwood) 2016;35:630–6. https://doi.org/10.
Handbook of Systematic Reviews of Interventions Version 5.1.0 [updated 1377/hlthaff.2015.1376.
March 2011], The Cochrane Collaboration. Available at: www.coc 49 Montgomery AA, Harding J, Fahey T. Shared decision making in
hranehandbook.org. Accessed on 10 June 2017. hypertension: the impact of patient preferences on treatment choice.
37 Schwitzer G. Pollution of health news: time to drain the swamp. Fam Pract 2001;18:309–13.
BMJ 2017;356:j1262. 50 Say RE, Thomson R. The importance of patient preferences in
38 Walsh-Childers K, Braddock J, Rabaza C, Schwitzer G. One step treatment decisions–challenges for doctors. BMJ 2003;327:542–5.
forward, one step back: changes in news coverage of medical 51 Braddock CH 3rd, Fihn SD, Levinson W, Jonsen AR, Pearlman RA.
interventions. Health Commun 2016:1–14. https://doi.org/10.1080/ How doctors and patients discuss routine clinical decisions.
10410236.2016.1250706. Informed decision making in the outpatient setting. J Gen Intern Med
39 Schwitzer G. Trying to drink from a fire hose: too much of the wrong 1997;12:339–45.
kind of health care news. Trends Pharmacol Sci 2015;36:623–7. 52 Braddock CH 3rd, Edwards KA, Hasenberg NM, Laidley TL,
40 Schwitzer G. How can journalists do a better job reporting on the Levinson W. Informed decision making in outpatient practice: time
principles of shared decision making? Chapter 41. In: Elwyn G, to get back to basics. JAMA 1999;282:2313–20.
Edwards A, Thompson R, editors. Shared Decision Making in Health 53 Glasziou P, Burls A, Gilbert R. Evidence based medicine and the
Care: Achieving Evidence-Based Patient Choice, 3rd edn. Oxford: medical curriculum. BMJ 2008;337:a1253. https://doi.org/10.1136/
Oxford University Press; 2016: 270–6. bmj.a1253.
41 Mulley AG, Trimble C, Elwyn G. Stop the silent misdiagnosis: 54 Hoffmann TC, Del Mar C. Clinicians’ expectations of the benefits
patients’ preferences matter. BMJ 2012;345:e6572. https://doi.org/ and harms of treatments, screening, and tests: a systematic review.
10.1136/bmj.e6572. JAMA Intern Med 2017;177:407–19. https://doi.org/10.1001/jamainte
42 King JS, Moulton BW. Rethinking informed consent: the case for rnmed.2016.8254.
shared medical decision-making. Am J Law Med 2006;32:429–501. 55 Dizon JM, Grimmer-Somers K. Complex interventions required to
43 Makoul G, Clayman ML. An integrative model of shared decision- comprehensively educate allied health practitioners on evidence-
making in medical encounters. Patient Educ Couns 2006;60:301–12. based practice. Adv Med Educ Pract 2011;2:8. https://doi.org/10.
44 Elwyn G, Frosch D, Thomson R, Joseph-Williams N, Lloyd A, 2147/AMEP.S19767.
Kinnersley P et al. Shared decision-making: a model for clinical 56 Fischer F, Lange K, Klose K, Greiner W, Kraemer A Barriers and
practice. J Gen Intern Med 2012;27:1361–7. strategies in guideline implementation – a scoping review.
45 Stacey D, Legare F, Col NF, Bennett CL, Barry MJ, Eden KB et al. Healthcare (Basel) 2016;4: pii: E36. https://doi.org/10.3390/healthcare
Decision aids for people facing health treatment or screening 4030036.
decisions. Cochrane Database Syst Rev 2014;(1). Art. No.: CD001431. 57 Greenhalgh T, Snow R, Ryan S, Rees S, Salisbury H. Six ‘biases’
https://doi.org/10.1002/14651858.cd001431.pub4. against patients and carers in evidence-based medicine. BMC Med
46 Montori VM, LeBlanc A, Buchholz A, Stilwell DL, Tsapas A. Basing 2015;1:200. https://doi.org/10.1186/s12916-015-0437-x.
information on comprehensive, critically appraised, and up-to-date 58 PLoS Medicine Editors. False hopes, unwarranted fears: the trouble
syntheses of the scientific evidence: a quality dimension of the with medical news stories. PLoS Med 2008;5:e118.
International Patient Decision Aid Standards. BMC Med Inform Decis 59 Shaughnessy AF, Gupta PS, Erlich DR, Slawson DC. Ability of an
Mak 2013;13(Suppl 2):S5. https://doi.org/10.1186/1472-6947-13-S2- information mastery curriculum to improve residents’ skills and
S5. attitudes. Fam Med 2012;44:259–64.
47 Elwyn G, Scholl I, Tietbohl C, Mann M, Edwards AGK, Clay C et al. 60 Coomarasamy A, Khan KS. What is the evidence that postgraduate
“Many miles to go . . .”: a systematic review of the implementation teaching in evidence based medicine changes anything? A
of patient decision support interventions into routine clinical systematic review. BMJ 2004;329:1017.
practice. BMC Med Inform Decis Mak 2013;13(Suppl 2):S14. https://
doi.org/10.1186/1472-6947-13-S2-S14.
48 Sepucha KR, Simmons LH, Barry MJ, Edgman-Levitan S, Licurse
AM, Chaguturu SK. Ten years, forty decision aids, and thousands of

8 ª 2017 Stichting European Society for Clinical Investigation Journal Foundation