Annals of Internal Medicine RESEARCH AND REPORTING METHODS

A Reporting Tool for Practice Guidelines in Health Care: The RIGHT

Statement
Yaolong Chen, PhD, MMed; Kehu Yang, MMed*; Ana Marušić, MD, PhD; Amir Qaseem, MD, PhD, MHA; Joerg J. Meerpohl, MD;
Signe Flottorp, MD, PhD; Elie A. Akl, MD, MPH, PhD; Holger J. Schünemann, MD, PhD; Edwin S.Y. Chan, PhD;
Yngve Falck-Ytter, MD; Faruque Ahmed, PhD; Sarah Barber, PhD; Chiehfeng Chen, MD, MPH, PhD; Mingming Zhang, MSc;
Bin Xu, MD; Jinhui Tian, PhD; Fujian Song, PhD; Hongcai Shang, MD, PhD; Kun Tang, PhD; Qi Wang, MMed; and
Susan L. Norris, MD, MPH, MSc*; for the RIGHT (Reporting Items for Practice Guidelines in Healthcare) Working Group†

The quality of reporting practice guidelines is often poor, and ance (items 16 and 17), funding and declaration and manage-
there is no widely accepted guidance or standards for such re- ment of interests (items 18 and 19), and other information (items
porting in health care. The international RIGHT (Reporting Items 20 to 22). The RIGHT checklist can assist developers in reporting
for practice Guidelines in HealThcare) Working Group was es- guidelines, support journal editors and peer reviewers when
tablished to address this gap. The group followed an existing considering guideline reports, and help health care practitioners
framework for developing guidelines for health research report- understand and implement a guideline.
ing and the EQUATOR (Enhancing the QUAlity and Transpar- Ann Intern Med. doi:10.7326/M16-1565 www.annals.org
ency Of health Research) Network approach. It developed a For author affiliations, see end of text.
checklist and an explanation and elaboration statement. The This article was published at www.annals.org on 22 November 2016.
RIGHT checklist includes 22 items that are considered essential * Corresponding authors.
for good reporting of practice guidelines: basic information † Members of the RIGHT Working Group are listed in Appendix 1 (available
(items 1 to 4), background (items 5 to 9), evidence (items 10 to at www.annals.org); their contributions are listed in Appendix 2 (available at
12), recommendations (items 13 to 15), review and quality assur- www.annals.org).

C lear, explicit, and transparent practice guidelines

enable health care practitioners, health administra-
tors, program managers, and the public to understand
Guidelines in HealThcare) checklist—focusing on the es-
sential items for reporting guidelines. Development of
this checklist followed the framework for health re-
and implement recommendations that may positively search reporting guidelines (9). We registered the proj-
affect patients and various populations (1). However, ect in the EQUATOR (Enhancing the QUAlity and Trans-
the quality of reporting practice guidelines seems to be parency Of health Research) Library (10). The RIGHT
low (2) and current tools to address this problem are Working Group drafted the project proposal, gener-
outdated or narrow or combine reporting and quality ated suggested items, recruited Delphi panelists, de-
assessment in a single instrument. The Conference on signed the questionnaires for the Delphi survey, and
Guideline Standardization published a checklist for re- drafted the final report. The Delphi group reviewed the
porting clinical practice guidelines (last updated in proposal, participated in 3 rounds of Delphi surveys,
2003) that focuses mainly on clinical medicine and thus came to consensus on the items included in the final
may not be directly applicable to public health or to checklist, and reviewed the final manuscript.
other types of guidelines (3). The AGREE (Appraisal of The RIGHT Working Group implemented a 4-step
Guidelines for REsearch and Evaluation) instrument was approach to generate potential items for the checklist.
developed for both quality assessment and reporting, First, the group reviewed 10 representative reporting
although it is widely regarded as an evaluation tool (4, guidelines highlighted in the EQUATOR Library to de-
5). Multifunction tools may not be optimal and must be termine how they generated potential items (11). These
distinguished from tools that address reporting and guidelines encompassed a wide variety of reporting
those that assess methodological quality because they tools, including those for randomized, controlled trials;
differ in purpose, structure, and content (6). Recently, diagnostic studies; observational studies; animal re-
the AGREE Next Steps Consortium published the search; economic evaluation; and systematic reviews.
AGREE reporting checklist based on the AGREE instru- One tool generated items based on a systematic review
ment (7, 8); however, this checklist is limited to items (12), whereas the others used surveys, group meetings,
derived from the original tool, was developed by a literature reviews, or combined approaches (13–21).
small group of researchers, and does not provide a de- Second, we conducted a comprehensive search of
tailed explanation or guidance about how to use it. handbooks and other documents to identify standards
or tools for guideline reporting (Appendix 3, Appendix
Figure, and Appendix Table, available at www.annals
DEVELOPMENT OF THE CHECKLIST
A multidisciplinary international team that included
policymakers, methodologists, epidemiologists, clini- See also:
cians, editors, and consumer representatives from 12
countries across Asia, Africa, Europe, Oceania, and Web-Only
North America was established in 2013. It aimed to de-
Supplement
velop a tool—the RIGHT (Reporting Items for practice
www.annals.org Annals of Internal Medicine 1

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016

 RESEARCH AND REPORTING METHODS The RIGHT Statement

.org). Third, 2 subgroups, each with 2 experienced in- cation style; and most important, the end users' needs.
vestigators, independently extracted potential checklist We recommend against deriving a score from the
items from all documents identified in the first 2 steps. checklist: the items may not be equally weighted, and
Last, the whole group held a face-to-face meeting to scores have been shown to be problematic in research
aggregate all potential items and remove duplicates. synthesis (25, 26).
After further discussion, 48 items were included in the We emphasize that the RIGHT checklist was not de-
initial list of potential items. Readers can obtain the veloped as a tool for assessing the quality of published
search results and initial list of items from the RIGHT practice guidelines—those tools, such as the AGREE in-
Web site (22). strument (27) and others (28), exist elsewhere. Rather,
For the Delphi survey, we recruited 17 persons with the RIGHT checklist is intended to complement those
experience in the development of practice or reporting tools. It also was not created as guidance for develop-
guidelines. These individuals encompassed a broad ing guidelines. Many handbooks exist for this purpose,
range of disciplines and diverse geographic represen- along with the Guidelines International Network–
tation. The Delphi technique followed the recommen- McMaster Guideline Development Checklist, a practical
dations of Murphy and colleagues (23) and Sinha and tool for guideline development supported by learning
coworkers (24) and included 3 rounds of e-mail– based resources (29). Readers should carefully select a tool
surveys. Panelists rated each item on a scale of 1 (not according to their specific needs.
important) to 5 (very important), suggested new items, The RIGHT checklist differs from the new AGREE
and provided comments that were circulated in subse- reporting checklist (8) in several important ways. First,
quent rounds. All panelists were asked to disclose any the structure of the AGREE reporting checklist follows
conflicts of interest before beginning the Delphi survey. the domains of AGREE II in scope and purpose, stake-
The response rate was 100% for all 3 rounds of the holder involvement, rigor of development, clarity of
Delphi process. presentation, applicability, and editorial indepen-
This study was funded by National Natural Science dence. In contrast, the RIGHT checklist emulates the
Foundation of China, which had no role in the study approach used by other reporting guidance state-
design, data collection and analysis, preparation of the ments, such as CONSORT (Consolidated Standards of
manuscript, or decision to publish the manuscript. Reporting Trials) (15) and PRISMA (Preferred Reporting
Items for Systematic reviews and Meta-Analyses) (13),
and orders items as the developer and reader would
CHECKLIST DESCRIPTION encounter them. For example, the RIGHT checklist
The RIGHT checklist consists of 22 items that we starts with the title, then the executive summary. Sec-
consider essential for good reporting of practice guide- ond, it includes important items that were not con-
lines (Table). These items encompass the following do- tained in the AGREE reporting checklist and should be
mains: basic information (items 1 to 4), background reported in a guideline: quality assurance, access, sug-
(items 5 to 9), evidence (items 10 to 12), recommenda- gestions for further research, and limitations of the
tions (items 13 to 15), review and quality assurance guideline. The RIGHT checklist highlights the impor-
(items 16 and 17), funding and declaration and man- tance of reporting PICO (population, intervention, com-
agement of interests (items 18 and 19), and other infor- parator, outcomes) questions and the quality of the
mation (items 20 to 22). The RIGHT explanation and body of evidence and includes 7 subitems on the for-
elaboration statement (Supplement, available at www mulation of recommendations from evidence. Finally,
.annals.org) provides readers with a comprehensive ex- the RIGHT explanation and elaboration statement (Sup-
planation and rationale, as well as examples of good plement) provides detailed information and examples,
reporting for each item in the checklist. which the AGREE reporting tool lacks.
Endorsement and implementation of reporting
guidelines may help reduce wasteful research and in-
DISCUSSION crease the potential effect of research on health (30).
The RIGHT checklist can assist guideline develop- We plan to use many approaches to promote the
ers in reporting guidelines, support journal editors and RIGHT checklist, such as asking authors of international
peer reviewers when considering guideline reports, guideline handbooks to add the checklist to new ver-
and help health care practitioners understand and im- sions of their handbooks, contacting the editors of the
plement a guideline. The checklist is useful for clinical core clinical journals in MEDLINE (www.nlm.nih.gov
practice guidelines and persons in public health and /bsd/aim.html) to elicit their support and encourage
other health care fields. It provides users and evalua- them to endorse the checklist, and informing guideline
tors a clear, explicit description of the processes and developers at international and national agencies and
procedures used to develop a guideline and access to professional societies about the RIGHT project.
the evidence used to formulate each recommendation. We followed an explicit, transparent, and docu-
The RIGHT checklist does not prescribe a specific mented process for developing the RIGHT checklist
format for reporting guidelines. Rather, each item and provide an accompanying explanation and elabo-
should be clearly presented and sufficiently detailed ration statement (Supplement). Persons from key inter-
somewhere in the guideline. For each item, order and national organizations and institutions that focus
format depend on the developer's preferences; publi- on development and implementation of guidelines
2 Annals of Internal Medicine www.annals.org

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016

 The RIGHT Statement RESEARCH AND REPORTING METHODS

Table. RIGHT Checklist

Section/Topic Number Item

Basic information
Title/subtitle 1a Identify the report as a guideline, that is, with “guideline(s)” or
“recommendation(s)” in the title.
1b Describe the year of publication of the guideline.
1c Describe the focus of the guideline, such as screening, diagnosis, treatment,
management, prevention, or others.
Executive summary 2 Provide a summary of the recommendations contained in the guideline.
Abbreviations and acronyms 3 Define new or key terms, and provide a list of abbreviations and acronyms if
applicable.
Corresponding developer 4 Identify at least 1 corresponding developer or author who can be contacted about
the guideline.

Background
Brief description of the health problem(s) 5 Describe the basic epidemiology of the problem, such as the
prevalence/incidence, morbidity, mortality, and burden (including financial)
resulting from the problem.
Aim(s) of the guideline and specific objectives 6 Describe the aim(s) of the guideline and specific objectives, such as
improvements in health indicators (e.g., mortality and disease prevalence),
quality of life, or cost savings.
Target population(s) 7a Describe the primary population(s) that is affected by the recommendation(s) in
the guideline.
7b Describe any subgroups that are given special consideration in the guideline.
End users and settings 8a Describe the intended primary users of the guideline (such as primary care
providers, clinical specialists, public health practitioners, program managers,
and policymakers) and other potential users of the guideline.
8b Describe the setting(s) for which the guideline is intended, such as primary care,
low- and middle-income countries, or inpatient facilities.
Guideline development groups 9a Describe how all contributors to the guideline development were selected and
their roles and responsibilities (e.g., steering group, guideline panel, external
reviewers, systematic review team, and methodologists).
9b List all individuals involved in developing the guideline, including their title,
role(s), and institutional affiliation(s).

Evidence
Health care questions 10a State the key questions that were the basis for the recommendations in PICO
(population, intervention, comparator, and outcome) or other format as
appropriate.
10b Indicate how the outcomes were selected and sorted.
Systematic reviews 11a Indicate whether the guideline is based on new systematic reviews done
specifically for this guideline or whether existing systematic reviews were used.
11b If the guideline developers used existing systematic reviews, reference these and
describe how those reviews were identified and assessed (provide the search
strategies and the selection criteria, and describe how the risk of bias was
evaluated) and whether they were updated.
Assessment of the certainty of the body of 12 Describe the approach used to assess the certainty of the body of evidence.
evidence

Recommendations
Recommendations 13a Provide clear, precise, and actionable recommendations.
13b Present separate recommendations for important subgroups if the evidence
suggests that there are important differences in factors influencing
recommendations, particularly the balance of benefits and harms across
subgroups.
13c Indicate the strength of recommendations and the certainty of the supporting
evidence.
Rationale/explanation for recommendations 14a Describe whether values and preferences of the target population(s) were
considered in the formulation of each recommendation. If yes, describe the
approaches and methods used to elicit or identify these values and
preferences. If values and preferences were not considered, provide an
explanation.
14b Describe whether cost and resource implications were considered in the
formulation of recommendations. If yes, describe the specific approaches and
methods used (such as cost-effectiveness analysis) and summarize the results. If
resource issues were not considered, provide an explanation.
14c Describe other factors taken into consideration when formulating the
recommendations, such as equity, feasibility, and acceptability.
Evidence to decision processes 15 Describe the processes and approaches used by the guideline development
group to make decisions, particularly the formulation of recommendations
(such as how consensus was defined and achieved and whether voting was
used).

Review and quality assurance

External review 16 Indicate whether the draft guideline underwent independent review and, if so,
how this was executed and the comments considered and addressed.
www.annals.org
Quality assurance 17 Indicate whether Annals of Internal
the guideline Medicine to a quality assurance process. If
was subjected 3
yes, describe the process.
Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016 Continued on following page
 RESEARCH AND REPORTING METHODS The RIGHT Statement

Table—Continued

Section/Topic Number Item

Funding and declaration and management of
interests
Funding source(s) and role(s) of the funder 18a Describe the specific sources of funding for all stages of guideline development.
18b Describe the role of funder(s) in the different stages of guideline development
and in the dissemination and implementation of the recommendations.
Declaration and management of interests 19a Describe what types of conflicts (financial and nonfinancial) were relevant to
guideline development.
19b Describe how conflicts of interest were evaluated and managed and how users of
the guideline can access the declarations.

Other information
Access 20 Describe where the guideline, its appendices, and other related documents can
be accessed.
Suggestions for further research 21 Describe the gaps in the evidence and/or provide suggestions for future research.
Limitations of the guideline 22 Describe any limitations in the guideline development process (such as the
development groups were not multidisciplinary or patients' values and
preferences were not sought), and indicate how these limitations might have
affected the validity of the recommendations.
RIGHT = Reporting Items for practice Guidelines in HealThcare.

contributed to this work, including the EQUATOR Net- public of Congo; Taipei Medical University–School of Medi-
work; Guidelines International Network; GRADE (Grad- cine, Taipei, Taiwan; Cochrane China, Sichuan, China; Nanjing
ing of Recommendations Assessment, Development University of Chinese Medicine, Nanjing, China; University of
and Evaluation) Working Group; AGREE Collaboration; East Anglia, Norwich, United Kingdom; Dongzhimen Hospital
and Cochrane Collaboration. The draft checklist and of Beijing University of Chinese Medicine and Peking Univer-
explanation and elaboration statement had extensive sity, Beijing, China; and World Health Organization, Geneva,
peer review by experts in guideline development with Switzerland.
diverse perspectives. We may have missed important
items when we developed our initial list of items, but Disclaimer: The findings and conclusions in this article are
we made every effort to minimize this possibility by ex- those of the authors and do not necessarily represent the
views of the World Health Organization or the Centers for
amining many guidance documents and manuals pro-
Disease Control and Prevention.
duced by guideline developers and consulting a broad
range of experts in this field.
Acknowledgment: The authors thank the persons who re-
The RIGHT checklist is available in English, Ger-
sponded to the Delphi survey for their thoughtful comments.
man, Croatian, Japanese, Korean, and simplified and
traditional Chinese; we encourage groups to make ad-
Grant Support: By National Natural Science Foundation of
ditional translations. We plan to develop RIGHT exten-
China (grant 81503459; Dr. Chen), China Fundamental Re-
sions, including RIGHT-P (for guideline proposals), search Funds for the Central Universities (grant 2016LZU-
RIGHT-COI (for conflicts of interest), and RIGHT-A (for JBZX159; Dr. Chen and Prof. Yang), the Open Fund of the Key
acupuncture). We ask persons who aim to develop re- Laboratory of Evidence-Based Medicine and Knowledge
lated standards or create translations to contact the Translation of Gansu Province (grant EBM1305 for the RIGHT
corresponding authors of this paper to coordinate ef- project), and Croatian Science Foundation (grant IP-2014-09-
forts and avoid duplication. 7672; Dr. Marušić).
Like any other reporting standard, the RIGHT
checklist is an evolving document that needs continual Disclosures: Dr. Meerpohl is a member of the GRADE Work-
assessment, improvement, and updating. We will revise ing Group and a member of the GRADE guidance committee.
the checklist in the future based on user feedback, re- Dr. Flottorp is a member of the GRADE Working Group and
sults of formal and informal evaluations, and new stud- the GRADE guidance committee. Dr. Akl is a member of the
ies on guideline reporting methods. We encourage us- GRADE Working Group. Dr. Schünemann is the co-chair of
ers to submit their comments via the RIGHT Web site. the GRADE Working Group and the lead author of the Guide-
lines International Network–McMaster Guideline Develop-
From Lanzhou University, Lanzhou, Gansu, China; University ment Checklist. Dr. Chan reports other part-time salary sup-
of Split School of Medicine, Split, Croatia; American College port from the Singapore Ministry of Health outside the
of Physicians, Philadelphia, Pennsylvania; Paris–Sorbonne Uni- submitted work. Dr. Falck-Ytter is a member of the GRADE
versity, Paris, France; Norwegian Institute of Public Health, Working Group. Authors not named here have disclosed no
Oslo, Norway; American University of Beirut, Beirut, Lebanon; conflicts of interest. Disclosures can also be viewed at
McMaster University, Hamilton, Ontario, Canada; Cochrane www.acponline.org/authors/icmje/ConflictOfInterestForms
Singapore, Biopolis, Singapore; Louis Stokes Cleveland Vet- .do?msNum=M16-1565.
erans Affairs Medical Center, Cleveland, Ohio; Centers for
Disease Control and Prevention, Atlanta, Georgia; World Requests for Single Reprints: Yaolong Chen, PhD, MMed,
Health Organization Regional Office for Africa, Brazzaville, Re- Lanzhou University, Donggang West Road, Chengguan Dis-
4 Annals of Internal Medicine www.annals.org

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016

 The RIGHT Statement RESEARCH AND REPORTING METHODS
trict, Lanzhou, Gansu 730000, China; e-mail, chenyaolong randomised trials. Ann Intern Med. 2010;152:726-32. [PMID:
@lzu.edu.cn. 20335313] doi:10.7326/0003-4819-152-11-201006010-00232
16. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Van-
Current author addresses and author contributions are avail- denbroucke JP; STROBE Initiative. Strengthening the Reporting of
Observational Studies in Epidemiology (STROBE) statement: guide-
able at www.annals.org.
lines for reporting observational studies. Ann Intern Med. 2007;147:
573-7. [PMID: 17938396]
17. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting
References qualitative research (COREQ): a 32-item checklist for interviews and
1. Oxman AD, Schünemann HJ, Fretheim A. Improving the use of focus groups. Int J Qual Health Care. 2007;19:349-57. [PMID:
research evidence in guideline development: 14. Reporting guide- 17872937]
lines. Health Res Policy Syst. 2006;4:26. [PMID: 17156458] 18. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP,
2. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guide- Irwig LM, et al; Standards for Reporting of Diagnostic Accuracy. To-
lines developed by specialty societies: the need for a critical ap- wards complete and accurate reporting of studies of diagnostic ac-
praisal. Lancet. 2000;355:103-6. [PMID: 10675167] curacy: the STARD initiative. Ann Intern Med. 2003;138:40-4. [PMID:
3. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, 12513043]
Deshpande AM. Standardized reporting of clinical practice guide- 19. Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney SE;
lines: a proposal from the Conference on Guideline Standardization. SQUIRE development group. Publication guidelines for quality im-
Ann Intern Med. 2003;139:493-8. [PMID: 13679327] provement studies in health care: evolution of the SQUIRE project.
4. Oxman AD, Schünemann HJ, Fretheim A. Improving the use of BMJ. 2009;338:a3152. [PMID: 19153129] doi:10.1136/bmj.a3152
research evidence in guideline development: 16. Evaluation. Health 20. Husereau D, Drummond M, Petrou S, Carswell C, Moher D,
Res Policy Syst. 2006;4:28. [PMID: 17156460] Greenberg D, et al; CHEERS Task Force. Consolidated Health Eco-
5. Wilson KC, Irwin RS, File TM Jr, Schünemann HJ, Guyatt GH, Rabe nomic Evaluation Reporting Standards (CHEERS) statement. BMJ.
KF; ATS/ERS Ad Hoc Committee on Integrating and Coordinating 2013;346:f1049. [PMID: 23529982] doi:10.1136/bmj.f1049
Efforts in COPD Guideline Development. Reporting and publishing 21. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Im-
guidelines: article 12 in Integrating and coordinating efforts in COPD proving bioscience research reporting: the ARRIVE guidelines for re-
guideline development. An official ATS/ERS workshop report. Proc porting animal research. J Pharmacol Pharmacother. 2010;1:94-9.
Am Thorac Soc. 2012;9:293-7. [PMID: 23256173] doi:10.1513/pats [PMID: 21350617] doi:10.4103/0976-500X.72351
.201208-065ST 22. RIGHT Statement. About RIGHT. 2014. Accessed at www.right
6. Huwiler-Müntener K, Jüni P, Junker C, Egger M. Quality of report- -statement.org on 9 July 2015.
ing of randomized trials as a measure of methodologic quality. 23. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF,
JAMA. 2002;287:2801-4. [PMID: 12038917] Askham J, et al. Consensus development methods, and their use in
7. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, clinical guideline development. Health Technol Assess. 1998;2:1-88.
Feder G, et al; AGREE Next Steps Consortium. AGREE II: advancing [PMID: 9561895]
guideline development, reporting and evaluation in health care. 24. Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to
CMAJ. 2010;182:E839-42. [PMID: 20603348] doi:10.1503/cmaj determine which outcomes to measure in clinical trials: recommen-
.090449 dations for the future based on a systematic review of existing stud-
8. Brouwers MC, Kerkvliet K, Spithoff K; AGREE Next Steps Consor- ies. PLoS Med. 2011;8:e1000393. [PMID: 21283604] doi:10.1371
tium. The AGREE Reporting Checklist: a tool to improve reporting of /journal.pmed.1000393
clinical practice guidelines. BMJ. 2016;352:i1152. [PMID: 26957104] 25. Greenland S, O’Rourke K. On the bias produced by quality
doi:10.1136/bmj.i1152 scores in meta-analysis, and a hierarchical view of proposed solu-
9. Moher D, Schulz KF, Simera I, Altman DG. Guidance for develop- tions. Biostatistics. 2001;2:463-71. [PMID: 12933636]
ers of health research reporting guidelines. PLoS Med. 2010;7: 26. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the
e1000217. [PMID: 20169112] doi:10.1371/journal.pmed.1000217 quality of clinical trials for meta-analysis. JAMA. 1999;282:1054-60.
10. EQUATOR Network. Reporting items for guidelines in health sys- [PMID: 10493204]
tems. 2014. Accessed at www.equator-network.org/library/reporting 27. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F,
-guidelines-under-development on 9 July 2015. Feder G, et al; AGREE Next Steps Consortium. The Global Rating
11. EQUATOR Network. Reporting guidelines for main study types. Scale complements the AGREE II in advancing the quality of practice
2014. Accessed at www.equator-network.org/library on 9 July 2015. guidelines. J Clin Epidemiol. 2012;65:526-34. [PMID: 22189163] doi:
12. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, 10.1016/j.jclinepi.2011.10.008
Krleža-Jerić K, et al. SPIRIT 2013 statement: defining standard pro- 28. Grimmer K, Dizon JM, Milanese S, King E, Beaton K, Thorpe O,
tocol items for clinical trials. Ann Intern Med. 2013;158:200-7. [PMID: et al. Efficient clinical evaluation of guideline quality: development
23295957] doi:10.7326/0003-4819-158-3-201302050-00583 and testing of a new tool. BMC Med Res Methodol. 2014;14:63.
13. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Pre- [PMID: 24885893] doi:10.1186/1471-2288-14-63
ferred reporting items for systematic reviews and meta-analyses: the 29. Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M,
PRISMA statement. Ann Intern Med. 2009;151:264-9. [PMID: Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development
19622511] of a comprehensive checklist for a successful guideline enterprise.
14. Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D; CARE CMAJ. 2014;186:E123-42. [PMID: 24344144] doi:10.1503/cmaj
Group. The CARE guidelines: consensus-based clinical case report .131237
guideline development. J Clin Epidemiol. 2014;67:46-51. [PMID: 30. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S,
24035173] doi:10.1016/j.jclinepi.2013.08.003 et al. Reducing waste from incomplete or unusable reports of bio-
15. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT medical research. Lancet. 2014;383:267-76. [PMID: 24411647] doi:
2010 statement: updated guidelines for reporting parallel group 10.1016/S0140-6736(13)62228-X

www.annals.org Annals of Internal Medicine 5

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016

 Current Author Addresses: Drs. Y. Chen and Tian, Prof. Yang, Author Contributions: Conception and design: Y. Chen, K.
and Ms. Wang: 199 Donggang West Road, Chengguan Dis- Yang, A. Qaseem, M. Zhang, B. Xu, Q. Wang, S.L. Norris.
trict, Lanzhou, Gansu 730000, China. Analysis and interpretation of the data: Y. Chen, K. Yang, J.J.
Dr. Marušić: Department of Research in Biomedicine and Meerpohl, S. Flottorp, H.J. Schünemann, S. Barber, S.L.
Health, University of Split School of Medicine, Soltanska 2, Norris.
21000 Split, Croatia. Drafting of the article: Y. Chen, K. Yang, A. Qaseem, J.J. Meer-
Dr. Qaseem: American College of Physicians, 190 N. Indepen- pohl, E.S.Y. Chan, F. Ahmed, S.L. Norris.
dence Mall West, Philadelphia, PA 19106-1572. Critical revision of the article for important intellectual con-
Dr. Meerpohl: Paris–Sorbonne University, 1 Rue Victor Cousin, tent: Y. Chen, K. Yang, A. Marušić, A. Qaseem, J.J. Meerpohl,
S. Flottorp, E.A. Akl, H.J. Schünemann, E.S.Y. Chan, Y. Falck-
75230 Paris cedex 05, France.
Ytter, F. Ahmed, S. Barber, C. Chen, M. Zhang, F. Song, S.L.
Dr. Flottorp: Norwegian Institute of Public Health, PO Box
Norris.
4404, Nydalen, N-0403 Oslo, Norway.
Final approval of the article: Y. Chen, K. Yang, A. Marušić, A.
Dr. Akl: Department of Internal Medicine, American University
Qaseem, J.J. Meerpohl, S. Flottorp, E.A. Akl, H.J. Schüne-
of Beirut Medical Center, PO Box 11-0236, Riad-El-Solh, Bei- mann, E.S.Y. Chan, Y. Falck-Ytter, F. Ahmed, S. Barber, J. Tian,
rut 1107 2020, Lebanon. F. Song, K. Tang, Q. Wang, S.L. Norris.
Dr. Schünemann: Department of Clinical Epidemiology & Bio- Obtaining of funding: Y. Chen, K. Yang.
statistics, McMaster University Health Sciences Centre, Room Administrative, technical, or logistic support: Y. Chen, K.
2C161280, Main Street West, Hamilton, Ontario L8S 4K1, Yang, H. Shang, K. Tang, S.L. Norris.
Canada. Collection and assembly of data: Y. Chen, K. Yang, A. Marušić,
Dr. Chan: Singapore Clinical Research Institute, 31 Biopolis E.A. Akl, E.S.Y. Chan, Y. Falck-Ytter, F. Ahmed, C. Chen, J.
Way, Nanos 02-01, Singapore 138669. Tian, Q. Wang, S.L. Norris.
Dr. Falck-Ytter: Louis Stokes Cleveland Veterans Affairs Medi-
cal Center, 10701 East Boulevard, Cleveland, OH 44106.
Dr. Ahmed: Centers for Disease Control and Prevention, 1600
Clifton Road Northeast, Mailstop E-03, Atlanta, GA 30329. APPENDIX 1: MEMBERS OF THE RIGHT
Dr. Barber: World Health Organization, Regional Office for WORKING GROUP
Africa, Room 303, Cité du Djoué, PO Box 06, Brazzaville, Re- Members of the RIGHT Working Group who au-
public of Congo.
thored this work are: Yaolong Chen, PhD, MMed, Kehu
Dr. C. Chen: Taipei Medical University–School of Medicine,
Yang, MMed, Jinhui Tian, PhD, and Qi Wang, MMed
250 Wuxing Street, Taipei 11031, Taiwan.
Ms. Zhang: Chinese Evidence-Based Medicine Centre, West
(Lanzhou University, Lanzhou, Gansu, China); Susan L.
China Hospital, 37 Guoxue Xiang, Chengdu, Sichuan 610041, Norris, MD, MPH, MSc (World Health Organization, Ge-
China. neva, Switzerland); Ana Marušić, MD, PhD (University of
Dr. Xu: Nanjing University of Chinese Medicine, 43 Box, 138 Split, Split, Croatia); Amir Qaseem, MD, PhD, MHA
Xianlin Road, Nanjing, Jiangsu 210023, China. (American College of Physicians, Philadelphia, Pennsyl-
Dr. Song: Department of Population Health and Primary Care, vania); Joerg J. Meerpohl, MD (Paris–Sorbonne Univer-
Norwich Medical School, University of East Anglia, Norwich, sity, Paris, France); Signe Flottorp, MD, PhD (Norwe-
NR4 7TJ, United Kingdom. gian Institute of Public Health, Oslo, Norway); Elie A.
Dr. Shang: Dongzhimen Hospital of Beijing University of Chi- Akl, MD, MPH, PhD (American University of Beirut, Bei-
nese Medicine, Dongcheng District, 5 Hai Yun Cang, Beijing
rut, Lebanon); Holger J. Schünemann, MD, PhD (Mc-
100700, China.
Master University, Hamilton, Ontario, Canada); Edwin
Dr. Tang: Peking University Health Science Center, 38 Xue
Yuan Road, Haidian District, Beijing 100191, China.
S.Y. Chan, PhD (Cochrane Singapore, Biopolis, Singa-
Dr. Norris: World Health Organization, Avenue Appia 20, CH- pore); Yngve Falck-Ytter, MD (Case Western Reserve
1211, Geneva 27, Switzerland. University); Faruque Ahmed, PhD (Centers for Disease
Control and Prevention, Atlanta, Georgia); Sarah Bar-
ber, PhD (World Health Organization Regional Office
for Africa, Brazzaville, Republic of Congo); Chiehfeng
Chen, MD, MPH, PhD (Taipei Medical University–
Municipal Wan Fang Hospital, Taipei, Taiwan); Ming-
ming Zhang, MSc (Chinese Cochrane Centre, Sichuan,
China); Bin Xu, MD (Nanjing University of Chinese Med-
icine, Nanjing, China); Fujian Song, PhD (University of
East Anglia, Norwich, United Kingdom); Hongcai
Shang, MD, PhD (Dongzhimen Hospital of Beijing Uni-
versity of Chinese Medicine, Beijing, China); and Kun
Tang, PhD (Peking University, Beijing, China).
Nonauthor members of the RIGHT Working Group
are: Hui Li, MD, PhD (Guangdong Provincial Hospital of
Chinese Medicine, Guangzhou, Guangdong, China);
www.annals.org Annals of Internal Medicine

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016

 Yan Hu, MD, PhD, and Boheng Zhang, MD, PhD (Fudan #1 guideline methodolog*
University, Shanghai, China); Huan Shen, MD, PhD, and #2 guideline handbook
Li Jiang, MD, PhD (Peking University People's Hospital, #3 guideline manual
Beijing, China); Suodi Zhai, MD, PhD (Peking University #4 guideline toolkit
Third Hospital, Beijing, China); and Xufei Luo, MBBS, #5 guidance methodolog*
and Yanfang Ma, MBBS (Lanzhou University, Lanzhou, #6 guidance handbook
Gansu, China). #7 guidance manual
#8 guidance toolkit
#9 consensus methodolog*
#10 consensus handbook
APPENDIX 2: CONTRIBUTIONS #11 consensus manual
Drs. Y. Chen and Norris and Prof. Yang conceived #12 consensus toolkit
the RIGHT project and drafted the project proposal. #13 recommendation methodolog*
Drs. Marušić, Qaseem, Meerpohl, Flottorp, Akl, Chan, #14 recommendation handbook
Falck-Ytter, Ahmed, Barber, C. Chen, and Xu and Ms. #15 recommendation manual
Zhang were Delphi panelists and gave comments and #16 recommendation toolkit
suggestions on the draft item list. Drs. Y. Chen, Song, #17 standard methodolog*
and Tang; Prof. Yang; and Ms. Wang generated sug- #18 standard handbook
gested items, designed the questionnaires for the Del- #19 standard manual
phi survey, and did the statistical analysis. Drs. Y. Chen #20 standard toolkit
and Norris drafted the manuscript, and all authors crit- We perused the reference lists of all eligible hand-
ically reviewed and revised it for important intellectual books for additional materials not captured by the
content. Dr. Y. Chen is the guarantor of the manuscript aforementioned searches. In addition, we examined
and affirms that it is an honest, accurate, and transpar- the reference lists in the articles by Ansari and Rashid-
ent account of the study being reported. All authors ian (31) and Vernooij and colleagues (32), which were
approved the final version of this article. identified in our literature review.
Eligibility Criteria
Handbooks that provided guidance on the entire
development process of practice guidelines were in-
APPENDIX 3: METHODS AND RESULTS FOR cluded. Documents that were written by individuals,
SYSTEMATICALLY SEARCHING FOR HANDBOOKS were outdated versions that had been subsequently
ON GUIDELINE DEVELOPMENT updated, or were focused on specific aspects of guide-
Search Strategy line development (such as updating; systematic re-
We conducted a systematic search on 30 April views, or the GRADE process) were excluded. We in-
2014 in MEDLINE (via PubMed; from 1966 onward) us- cluded handbooks (English language only) if they
ing the following combination of free-text terms: contained a section on how to present, write, or report
#1 Clinical Practice Guideline*[tw] a guideline.
#2 Clinical guideline*[tiab] Handbook Selection
#3 Guideline*[ti] Two reviewers independently screened all identi-
#4 guidance*[ti] fied records (Q.W. and K.T.). Disagreements were re-
#5 consensus[ti] solved by consensus, and if necessary, with the help of
#6 recommendation*[ti] team leader (Y.C.). We ultimately included 30 hand-
#7 OR#1-#6 books on guideline development, two thirds of which
#8 methodolog*[tiab] contained sections on reporting.
#9 handbook*[tiab]
#10 manual*[tiab] Web-Only References
#11 toolkit*[tiab] 31. Ansari S, Rashidian A. Guidelines for guidelines: are they up to
the task? A comparative assessment of clinical practice guideline
#12 OR#8-#11 development handbooks. PLoS One. 2012;7:e49864. [PMID:
#13 #7 AND #12 23189167] doi:10.1371/journal.pone.0049864
#14 Practice Guideline[pt] 32. Vernooij RW, Sanabria AJ, Solà I, Alonso-Coello P, Martı́nez
Garcı́a L. Guidance for updating clinical practice guidelines: a sys-
#15 #13 NOT #14 tematic review of methodological handbooks. Implement Sci. 2014;
On 30 April 2014, we also performed a search us- 9:3. [PMID: 24383701] doi:10.1186/1748-5908-9-3
ing the Google search engine (Alphabet). We added
the following search terms individually in Google and
browsed the first 200 records for each term:
Annals of Internal Medicine www.annals.org

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016

 Appendix Figure. Characteristics of the included
handbooks.

Records collected by browsing

Records identified through
4000 Google records
PubMed (n = 3678)
(n = 4000)

References of
Handbooks included
all eligible
according to eligibility
handbooks
criteria (n = 27)
(n = 3)

Final included
handbooks (n = 30)

Handbooks with chapters, guidance, or

information about guideline
reporting (n = 20)

www.annals.org Annals of Internal Medicine

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016

 Appendix Table. Handbook Selection

Number Title Year Institution Reporting Guidance*

1 Guidelines for Clinical Practice: From 1992 U.S. Department of Health and Human Services None
Development to Use
2 A Guide to the Development, 1998 National Health Medical Research Council Full
Implementation and Evaluation of Clinical
Practice Guidelines
3 How to Develop Cost-Conscious Guidelines 2001 Health Technology Assessment National Health None
Service Research and Development Health
Technology Assessment Programme
4 Handbook for the Preparation of Explicit 2001 New Zealand Guidelines Group Full
Evidence-Based Clinical Practice
Guidelines
5 Developing a Methodology for Drawing Up 2002 Committee of Ministers of the Council of Europe None
Guidelines on Best Medical Practices
6 Guide for Guideline: A Guide for Clinical 2003 International Diabetes Federation None
Guideline Development
7 Handbook: Developing and Applying 2003 Regional Centre for Quality of Health Care Part
National Guidelines on Nutrition and
HIV/AIDS
8 Framework for Clinical Guideline 2004 European Region of the World Confederation for Full
Development in Physiotherapy Physical Therapy
9 Registered Nurses' Association of Ontario 2006 Nursing Best Practice Research Unit Full
Guideline Development Methodology
10 Evidence-Based Care Guideline 2006 Cincinnati Children's Hospital Medical Center None
Development and Update Process
11 Handbook on Clinical Practice Guideline 2007 Canadian Medical Association Full (refer to the
Development Conference on Guideline
Standardization)
12 U.S. Preventive Services Task Force 2008 Agency for Healthcare Research and Quality None
Procedure Manual
13 Clinical Practice Guideline Development 2009 World Stroke Organization Stroke Guideline Full
Handbook for Stroke Care Sub-Committee
14 Updating the Guideline Methodology of the 2009 Healthcare Infection Control Practices Advisory None
Healthcare Infection Control Practices Committee
Advisory Committee
15 Methodology Manual and Policies From the 2010 American College of Cardiology Foundation and Part
ACCF/AHA Task Force on Practice American Heart Association
Guidelines
16 Clinical Practice Guidelines Process Manual 2011 American International Health Alliance None
17 Clinical Practice Guideline Process Manual 2011 American Academy of Neurology Full
18 British HIV Association Guideline 2011 British HIV Association Full
Development Manual
19 SIGN 50: A Guideline Developer's 2011 Scottish Intercollegiate Guidelines Network Full
Handbook
20 WHO Handbook for Guideline Development 2012 World Health Organization Part
21 Concise Guidelines Series Handbook: A 2012 Royal College of Physicians Clinical Standards Full
Series of Evidence-Based Guidelines for Department
Clinical Management
22 Guidelines and Protocols Advisory 2012 Guidelines and Protocols Advisory Committee None
Committee Handbook: Clinical Practice British Columbia Medical Association/The
Guidelines and Protocols by the Medical Services Branch of the British
Guidelines and Protocols Advisory Columbia/Ministry of Health
Committee
23 Process and Methods Guides: The 2012 National Institute for Health and Care Excellence Full
Guidelines Manual
24 Clinical Practice Guideline Development 2013 American Academy of Otolaryngology–Head and Full
Manual, Third Edition: A Quality-Driven Neck Surgery
Approach for Translating Evidence into
Action
25 ADA Clinical Practice Guidelines Handbook 2013 American Dental Association Center for Full
Evidence-Based Dentistry
26 National Clinical Effectiveness Committee 2013 National Clinical Effectiveness Committee Full
Guideline Developers Manual Appendix
IV: NCEC Clinical Guideline Template
27 British Thoracic Society Standards of Care 2013 British Thoracic Society Full
Committee Guideline Production Manual
28 ASCO Guideline Procedures Manual 2014 The American Society of Clinical Oncology None
29 Manual for ESHRE Guideline Development 2014 European Society of Human Reproduction and Full
Embryology
30 KHA-CARI Guidelines Development Manual 2014 Board of Kidney Health Australia Full
* “None” indicates that the handbook does not contain any chapters or sections on reporting guidance. “Part” indicates that the handbook contains
several sentences or paragraphs on reporting guidance but not an entire chapter. “Full” indicates that the handbook contains at least 1 entire
chapter on reporting guidance.

Annals of Internal Medicine www.annals.org

Downloaded From: http://annals.org/ by a University of California San Diego User on 12/17/2016