Education and debate

Clinical guidelines
Developing guidelines
Paul G Shekelle, Steven H Woolf, Martin Eccles, Jeremy Grimshaw

The methods of guideline development should ensure This is the

that treating patients according to the guidelines will Summary points second in a
achieve the outcomes that are desired. This article series of four
presents a combination of the literature about articles on
Identifying and refining the subject area is the
guideline development and the results of our issues in the
first step in developing a guideline
combined experience in guideline development in development
North America and Britain. It considers the five steps Convening and running guideline development and use of
in the initial development of an evidence based guide- groups is the next step clinical
line. The dissemination, implementation, and evalua- guidelines
tion of practice guidelines will be discussed in the final On the basis of systematic reviews, the group
article in this series.1 assesses the evidence about the clinical question West Los Angeles
Veterans Affairs
or condition Medical Center
(111G), 11301
Identifying and refining the subject area This evidence is then translated into a Wilshire Blvd,
Los Angeles,
of a guideline recommendation within a clinical practice CA 90073, USA
guideline Paul G Shekelle,
Prioritising topics senior research
Guidelines can be developed for a wide range of associate, Health
The last step in guideline development is external Services Research and
subjects. Clinical areas can be concerned with
review of the guideline Development Service
conditions (abnormal uterine bleeding, coronary
Department of
artery disease) or procedures (hysterectomy, coronary Family Practice,
artery bypass surgery). Given the large number of Virginia
One method of defining the clinical question of Commonwealth
potential areas, some priority setting is needed to select
interest and also identifying the processes for which University, Fairfax,
an area for guideline development. Potential areas can Virginia 22033,
evidence needs to be collected and assessed is the con-
emerge from an assessment of the major causes of USA
struction of models or causal pathways.2 A causal path- Steven H Woolf
morbidity and mortality for a given population, uncer-
way is a diagram that illustrates the linkages between professor of family
tainty about the appropriateness of healthcare medicine
intervention(s) of interest and the intermediate, surro-
processes or evidence that they are effective in improv-
gate, and health outcomes that the interventions are Centre For Health
ing patient outcomes, or the need to conserve Services Research,
thought to influence. In designing the pathway, guide- University of
resources in providing care.
line developers make explicit the premises on which Newcastle upon
their assumptions of effectiveness are based and the Tyne, Newcastle
upon Tyne
Refining the subject area outcomes (benefits and harms) that they consider NE2 4AA
The topic for guideline development will usually need important. This identifies the specific questions that Martin Eccles,
to be refined before the evidence can be assessed in must be answered by the evidence to justify professor of clinical
effectiveness
order to answer exact questions. The usual way of conclusions of effectiveness and highlights gaps in the
evidence, for which future research is needed. Health Services
refining the topic is by a dialogue among clinicians, Research Unit,
patients, and the potential users or evaluators of the University of
guideline. Discussions about the scope of the guideline Aberdeen,
Running guideline development groups Aberdeen AB9 2ZD
will also take place within the guideline development
Jeremy Grimshaw
panel. Setting up a guideline development project professor of public
If the topic is not refined, the clinical condition or To successfully develop a guideline it may be necessary health
question may be too broad in scope. For example, a to convene more than one group. A project or Correspondence to:
guideline on the management of diabetes could cover management team could undertake the day to day Dr Shekelle
[email protected]
primary, secondary, and tertiary care elements of running of the work, such as the identification, synthe-
Series editors:
management and also multiple aspects of manage- sis, and interpretation of relevant evidence; the Martin Eccles,
ment, such as screening, diagnosis, dietary manage- convening and running of the guideline development Jeremy Grimshaw
ment, drug therapy, risk factor management, or groups; and the production of the resulting guidelines.
indications for referral to a consultant. Though all of Additional guideline development group(s) would BMJ 1999;318:593–6

these could legitimately be dealt with in a guideline, produce recommendations in the light of the evidence
the task of developing such a guideline would be con- or of its absence.
siderable; therefore a group needs to be clear which
areas are and are not within the scope of their Group membership and roles
activities. It is possible to develop guidelines that are Group members—Identifying stakeholders involves
both broad in scope and evidence based, but to do so identifying all the groups whose activities would be
usually requires considerable time and money, both of covered by the guideline or who have other legitimate
which are frequently underestimated by inexperi- reasons for having an input into the process. This is
enced developers of evidence based clinical practice important to ensure adequate discussion of the
guidelines. evidence (or its absence) when developing the recom-

BMJ VOLUME 318 27 FEBRUARY 1999 www.bmj.com 593

Education and debate

assess its potential applicability to the clinical ques-

tion under consideration, inspect the evidence for
susceptibility to bias, and extract and summarise the
findings.

What sort of evidence?

Identifying the clinical questions of interest will help
set the boundaries for admissible evidence (types of
study designs, year of publication, etc). For example,
questions of the efficacy of interventions usually mean
that randomised controlled trials should be sought,
while questions of risk usually mean that prospective
cohort studies should be sought.

Where to look for evidence?

MARK McCONNELL
The first step in gathering the evidence is to see if a
suitable, recent systematic review has already been
published. The Cochrane Library will also identify
relevant Cochrane review groups, which should also be
mendations in the guideline. When presented with the contacted to see if a review is in progress.
same evidence a single specialty group will reach If a current systematic review is not available, a
different conclusions than a multidisciplinary group— computer search of Medline and Embase is the usual
the specialty group will be systematically biased in starting point, using search strategies tailored to
favour of performing procedures in which it has a appropriate types of studies (though such strategies
vested interest.3 4 For example, the conclusions of a have been validated only for randomised controlled
group of vascular surgeons favoured the use of carotid trials9). For example, randomised controlled trials pro-
endarterectomy more than did a mixed group of vide the best evidence to answer questions about the
surgeons and medical specialists.5 Individuals’ biases effectiveness of treatments, whereas prospective
may be better balanced in multidisciplinary groups, cohort studies generally provide the best evidence for
and such balance may produce more valid guidelines. questions about risk. The Cochrane controlled trials
Ideally the group should have at least six but no more register (part of the Cochrane Library) contains
than 12-15 members; too few members limits adequate references to over 218 000 clinical trials that have
discussion and too many members makes effective been identified though database and hand searching;
functioning of the group difficult. Under certain it should be examined early on in any review process.
circumstances (for example, guidelines for broad clini- Checking references in articles will show additional
cal areas) it may be necessary to trade off full represen- relevant articles not identified by the computer search,
tation against the requirement of having a functional and having experts in the field examine the list of
group. articles helps ensure there are no obvious omissions.
Roles—Roles required within guideline develop- Additional search strategies, including searches for
ment groups are those of group member, group articles published in languages other than English,10–12
leader, specialist resource, technical support, and computer searches of specialised databases, hand
administrative support. Group members are invited to searching relevant journals, and searching for unpub-
participate as individuals working in their field; their lished material, will often yield additional studies, but
role is to develop recommendations for practice based the resources needed for such activities are con-
on the available evidence and their knowledge of the siderable. The cost effectiveness of various search
practicalities of clinical practice. strategies has not been established. It is best to match
The role of the group leader is both to ensure that the scope of the search strategy to the available
the group functions effectively (the group process) and resources.
that it achieves its aims (the group task). The process is
best moderated by someone familiar with (though not Assessing studies for relevance
necessarily an expert in) the management of the clini- Once studies have been identified, they are assessed
cal condition and the scientific literature, but who is not for relevance to the clinical questions of interest and
an advocate. He or she stimulates discussion and allows for bias.13 14 Screening for relevance is often possible
the group to identify where true agreement exists but from the abstract; it narrows the set of studies to those
does not inject his or her own opinion in the process.
This requires someone with both clinical skills and
group process skills. Using formal group processes Skills needed for guideline development
rather than informal ones in group meetings produces • Literature searching and retrieval
different and possibly better outcomes.6–8 • Epidemiology
• Biostatistics
• Health services research
Identifying and assessing the evidence • Clinical experts
Identifying and assessing the evidence is best done by • Group process experts
performing a systematic review. The purpose of a • Writing and editing
systematic review is to collect all available evidence,

594 BMJ VOLUME 318 27 FEBRUARY 1999 www.bmj.com

 Education and debate

needing a more detailed assessment. Using explicit

rather than implicit criteria should improve the Factors contributing to the process of deriving
reliability of the process. recommendations
• The nature of the evidence (for example, its
Summarising evidence susceptibility to bias)
Data are extracted from the relevant studies on the • The applicability of the evidence to the population
benefits, the harms, and (where applicable) the costs of of interest (its generalisability)
the interventions being considered. These data are • Costs
usually presented in a form that allows the designs and • Knowledge of the healthcare system
results of studies to be compared. Where appropriate, • Beliefs and values of the panel
meta-analysis can be used to summarise results of
multiple studies.
given illness, but more commonly the evidence needs
Categorising evidence to be interpreted into a clinical, public health, policy, or
Summarised evidence is categorised to reflect its payment context. Therefore within the guideline
susceptibility to bias. This is a shorthand method of development process a decision should be taken about
conveying specific aspects of the evidence to a reader how opinion will be both used and gathered.
of the guideline. A number of such “strength of
evidence” classification schemes exist, but empirical
Using and gathering opinion
evidence exists only for schemes that categorise
Opinion will be used to interpret evidence and also to
effectiveness studies by study design.15 16 The box shows
derive recommendations in the absence of evidence.
a simple scheme for classifying the evidence that
When evidence is being interpreted, opinion is needed
supports statements in practice guidelines and the
to assess issues such as the generalisability of
strength of the recommendations. Guideline develop-
evidence—for example, to what degree evidence from
ers should use a limited number of explicit criteria,
small randomised clinical trials or controlled observa-
incorporating criteria for which there is explicit
tional studies may be generalised, or to extrapolate
evidence.
results from a study in one population to the
population of interest in the guideline (extrapolating a
Translating evidence into a clinical study in a tertiary, academic medical centre to the
practice guideline community population of interest to potential users of
the guideline).
The evidence, once gathered, needs to be interpreted
Recommendations based solely on clinical judg-
(see box). Since conclusive evidence exists for relatively
ment and experience are likely to be more susceptible
few healthcare procedures, deriving recommendations
to bias and self interest. Therefore, after deciding what
solely in areas of strong evidence would lead to a
role expert opinion is to play, the next step is deciding
guideline of limited scope or applicability.17 This could
how to collect and assess expert opinion. There is cur-
be sufficient if, for example, the guideline is to recom-
rently no optimal method for this, but the process
mend the most strongly supported treatments for a
needs to be made as explicit as possible.

Resource implications and feasibility

Classification schemes
In addition to scientific evidence and the opinions of
Category of evidence: expert clinicians, practice guidelines must often take
Ia—evidence for meta-analysis of randomised account of the resource implications and feasibility of
controlled trials interventions. Judgments about whether the costs of
Ib—evidence from at least one randomised controlled
tests or treatments are reasonable depend on how cost
trial
IIa—evidence from at lease one controlled study effectiveness is defined and calculated, on the perspec-
without randomisation tive taken (for example, clinicians often view cost
IIb—evidence from at lease one other type of implications differently than would payers or society at
quasi-experimental study large), and on the resource constraints of the
III—evidence from non-experimental descriptive healthcare system (for example, cash limited public
studies, such as comparative studies, correlation systems versus private insurance based systems).
studies, and case-control studies
IV—evidence from expert committee reports or
Feasibility issues worth considering include the time,
opinions or clinical experience of respected skills, staff, and equipment necessary for the provider
authorities, or both to carry out the recommendations, and the ability of
patients and systems of care to implement them.
Strength of recommendation:
A—directly based on category I evidence
B—directly based on category II evidence or Grading recommendations
extrapolated recommendation from category I It is common to grade each recommendation in the
evidence guideline. Such information provides the user with an
C—directly based on category III evidence or
indication of the guideline development group’s confi-
extrapolated recommendation from category I or II
evidence dence that following the guideline will produce the
D—directly based on category IV evidence or desired health outcome. “Strength of recommen-
extrapolated recommendation from category I, II or dation” classification schemes (such as the one in the
III evidence box) range from simple to complex; no one scheme
has been shown to be superior. Given the factors that

BMJ VOLUME 318 27 FEBRUARY 1999 www.bmj.com 595

Education and debate

contribute to a recommendation, strong evidence does x A systematic review of the evidence should be at the
not always produce a strong recommendation, and the heart of every guideline; and
classification should allow for this. The classification is x The group assembled to translate the evidence into
probably best done by the group panel, using a demo- a guideline should be multidisciplinary.
cratic voting process after group discussion of the
strength of the evidence.
1 Feder G, Eccles M, Grol R, Griffiths C, Grimshaw J. Using clinical guide-
lines. BMJ (in press).
2 Woolf SH. An organized analytic framework for practice guideline devel-
opment: using the analytic logic as a guide for reviewing evidence, devel-
Reviewing and updating guidelines oping recommendations, and explaining the rationale. In: McCormick
KA, Moore SR, Siegel RA, eds. Methodology perspectives. Washington, DC:
Guidelines should receive external review to ensure US Department of Health and Human Services, Agency for Health Care
content validity, clarity, and applicability. External Policy and Research, 1994:105-13. (AHCPR publication No 95-0009.)
3 Kahan JP, Park RE, Leape LL, Bernstein SJ, Hilborne LH, Parker L, et al.
reviewers should cover three areas: people with exper- Variations by specialty in physician ratings of the appropriateness and
tise in clinical content, who can review the guideline to necessity of indications for procedures. Med Care 1996;34:512-23.
verify the completeness of the literature review and to 4 Coulter I, Adams A, Shekelle P. Impact of varying panel membership on
ratings of appropriateness in consensus panels—a comparison of a multi-
ensure clinical sensibility; experts in systematic reviews and single disciplinary panel. Health Serv Res 1995;30;577-91.
or guideline development, or both, who can review the 5 Leape LL, Park RE, Kahan JP, Brook RH. Group judgments of appropri-
ateness: the effect of panel composition. Quality Assur Health Care
method by which the guideline was developed; and 1992;4:151-9.
potential users of the guideline, who can judge its use- 6 Kosecoff JH, Kanouse DE, Rogers WH, McCloskey L, Winslow CM,
Brook RH. Effects of the National Institutes of Health consensus
fulness. In Britain there is a further review process development program on physician practice. JAMA 1987;258:2708-13.
whereby guidelines are appraised by an independent 7 Shekelle PG, Schriger DL. Evaluating the use of the appropriateness
unit to assess whether the NHS Executive can method in the agency for health care policy and research clinical practice
guideline development process. Health Serv Res 1996;31:453-68.
commend them to the NHS. 8 Shekelle PG, Kravitz RL, Beart J, Morger M, Wang M, Lee M. Are nonspe-
The guideline can be updated as soon as each piece cific practice guidelines potentially harmful? A randomized comparison
of the effect of nonspecific or specific guidelines on physician decision
of relevant new evidence is published, but it is better to making. Health Services Research (in press).
specify a date for updating the systematic review that 9 Cochrane Review Handbook. In: Cochrane Collaboration. Cochrane
Library. Issue 4. Oxford: Update Software, 1998.
underpins the guideline.
10 Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for
systematic reviews. BMJ 1994;309:1286.
11 Gregoire G, Derderian F, Le Lorier J. Selecting the language of the pub-
lications included in a meta-analysis: is there a Tower of Babel bias? J Clin
Conclusions Epidemiol 1995;48:159-63.
12 Egger M, Zellweger-Zohner T, Schneider M, Junker C, Lengeler C, Antes
New advances in understanding the science of G. Language bias in randomised controlled trials published in English
systematic reviews, the workings of groups of experts, and German. Lancet 1997;350:326-9.
13 Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias:
and the relation between guideline development and dimensions of methodological quality associated with estimates of treat-
implementation are all likely in the next three to five ment effects in controlled trials. JAMA 1995;273:408-12.
14 Moher D, Jones A, Cook D, Jadad AR, Moher M, Tugwell P, et al. Does
years.
quality of reports of randomised trials affect estimates of intervention
We believe that three principles will remain basic to efficacy reported in meta-analyses? Lancet 1998;352:609-13.
the development of valid and usable guidelines: 15 Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in
comparisons of therapy. I: medical. Stat Med 1989;8:441-54.
x The development of guidelines requires sufficient 16 Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in
resources in terms of people with a wide range of skills, comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
17 Shekelle P. Assessing the predictive validity of the RAND/UCLA appro-
including expert clinicians, health services researchers, priateness method criteria for performing carotid endarterectomy. Int J
and group process leaders and financial support; Technol Assess Health Care 1998;14:707-27.

A patient who changed my practice

Patients not protocols

“Dear Dr Elizabeth, “Why not leave me to toddle gently on in old age, I quite enjoy
“I am returning the Lipitor tablets as well as the Zocor. I take it, and fancy medicines just make me bad-tempered with
them about 3.0 p.m.; the first 2 days they did not have much everyone. When a stroke comes, it comes, I know its effects. If all
effect, but on Thursday I had some aches and pains and wobbles that is left is the enjoyment of sun on my eyelids, it is still
and on Friday 1 did not know where to put myself in the evening! enjoyment and death can be enjoyable too.
Alright, after a night’s sleep. Yours as a complete hedonist,”
“A year or so ago, I remember you saying that 6.5 wasn’t too (Name withheld at patient’s request; she is in her 70s)
bad a cholesterol level at my age, then you read something or I believe this letter to exemplify the essence of general practice.
discussed something with somebody and put me on Zocor, which How can the restraints of clinical governance take into account
did nothing. Please do not put me on anything else; I promise I patients like this?
will diet to cruelty point, even cutting out those 4 chocolates at
Elizabeth A McClure, general practitioner, Chester
Christmas and Easter. After all I got down from 9.5 to 7.0 entirely
on diet plus, importantly, dried garlic tablets, (which you are not We welcome articles up to 600 words on topics such as
allowed to prescribe). A memorable patient, A paper that changed my practice, My most
“When I have dropped down dead I promise I will not tell you unfortunate mistake, or any other piece conveying instruction,
off for neglect nor more importantly will my relatives. The pathos, or humour. If possible the article should be supplied on a
nifedipine keeps the angina from bothering me much (as long as disk. Permission is needed from the patient or a relative if an
I take life easy), the bendrofluazide helps my breathing, the identifiable patient is referred to. We also welcome contributions
aspirin thins my blood— so far it keeps going round alright. I for “Endpieces,” consisting of quotations of up to 80 words (but
really don’t want to enter any longevity stakes; so far, except for most are considerably shorter) from any source, ancient or
those first doses of angina, I have had pretty good health. modern, which have appealed to the reader.

596 BMJ VOLUME 318 27 FEBRUARY 1999 www.bmj.com