Can I trust this evidence?

Well-designed research can produce relatively

unbiased answers to clinical questions. Poorly
designed research can generate biased answers.

OVERVIE Readers of the clinical research literature need to

be able to discriminate between well-designed and
W poorly designed research.

As discussed in the previously, ideally the search

for evidence will yield a small number of studies. If
you have systematically sought out studies of the
type needed to answer your question then you can
begin the process of critical appraisal
A PROCESS FOR CRITICAL APPRAISAL OF EVIDENCE

Many physiotherapists experience a common frustration. When they consult the research literature for answers
to clinical questions they are confronted by a range of studies with very different conclusions.

Consider, for example, the findings that confront a physiotherapist who would like to know whether
acupuncture protects against exercise- induced asthma. One study, by Fung et al (1986) concluded ‘acupuncture
provided better protection against exercise-induced asthma than did sham acupuncture’. On the other hand,
Gruber et al (2002) concluded ‘acupuncture treatment offers no protection against exercise-induced
bronchoconstriction’.

These conclusions appear inconsistent. It seems implausible that both could be true. Situations like this, where
similar studies draw contradictory conclusions, often arise.
Why is the literature apparently so
inconsistent?

• There are several possible explanations.

1. First, there may be important differences between
studies in the type of patients included,
2. The way in which the intervention was administered,
and
3. The way in which outcomes were measured.
 Many studies are poorly designed and may therefore have
seriously biased conclusions.

Bias (an The findings of poorly designed studies and well-controlled

studies of the same interventions can differ very markedly
imporatnt cause
of Of the two studies of acupuncture for exercise-induced asthma
inconsistency) cited above, only the study by Gruber et al (2002) blinded the
subjects and assessors of outcomes

The inconsistency of the conclusions of these studies may arise

because the study by Gruber et al provides a relatively unbiased
estimate of the effects of acupuncture, while the study by Fung
et al (1986) may have been sub- ject to a range of biases.
 Methodologists have conducted numerous
How much surveys of the quality of published research and
the conclusion has almost always been that much
of the of the published research is of poor quality (see,
for example, Anyanwu & Treasure 2004,
published Kjaergard et al 2002, Dickinson et al 2000).

research is Many people who are not familiar with the

of high research process find it difficult to believe that
much of the published research is potentially
quality? seriously biased
How They imagine that research is usually carried
out by experts, that research reports are
much of peer-reviewed by people with
methodological expertise, and that research
the papers are therefore usually of a high
standard
published
research is The reality is that much of the clinical
research we read in journals is conducted by
of high people who have little or no training in
research design.
quality?
The quality of randomized trials

• A quantitative estimate of the

quality of randomized trials in
physiotherapy is provided by the
PEDro database. All trials on the
database are assessed according to
ten methodological criteria. A
methodological quality score is
generated by counting the number
of criteria that are satisfied.
• If it is true that a substantial proportion of the clinical research
published in journals is poorly designed and potentially misleading,
readers of clinical research must be able to distinguish between high
quality studies that potentially provide useful information for clinical
decision- making and low quality clinical research which is potentially
misleading.
This might appear to be too much to ask of readers. Surely, if many researchers
and journals reviewers cannot distinguish between high quality and low quality
research, it is unreasonable to expect readers of clinical trials to be able to do so

it is probably possible to use very simple checklists to distinguish coarsely

between high quality research and research that is likely to be biased.

One set of criteria that is of particular interest is the Delphi list of criteria for
assessing the quality of clinical trials, developed by Verhagen and colleagues
(Verhagen et al 1998a)
CRITICAL APPRAISAL OF EVIDENCE ABOUT THE
EFFECTS OF INTERVENTION

• Readers of clinical trials can ask three questions to discriminate coarsely

between those trials that are likely to be valid and those that are
potentially seriously biased.
1. Were treated and control groups comparable?
2. Was there complete or near complete follow-up?
3. Was there blinding to allocation of patients and assessors?
 we only expect to obtain unbiased estimates of the effects
of intervention from studies that compare outcomes in
1. Were treated and untreated groups.

treated and It is essential that the groups are comparable,

control
groups comparability can only be assured by randomly assigning
comparable subjects to groups.

? Matching’ of subjects in the treatment and control groups

cannot, on its own, ensure that the groups are comparable,
regardless of how diligently the matching is carried out.
The process of
Randomization
• Randomization is best achieved
by using a computer to generate
an allocation schedule.
• Alternatively, random allocation
schedules can be generated by
effectively random processes
like coin-tossing or the drawing
of lots
• Sometimes quasi-random
allocation procedures are used
The process of Randomization

Some studies match subjects and randomly allocate subjects to

groups. The technical term for this is stratified random allocation.

In general, stratified random allocation ensures more similarity

between groups, but usually only slightly more similarity, than would
occur with simple randomization
How do we know about Randomization in clinical
research?

• It is usually a very easy matter to determine if a clinical trial was randomized

or not. Reports of randomized trials will usually explain that subjects were
‘randomly allocated to groups’. This might appear in the title of the paper, or
in the abstract, or in the Methods section.
 True randomization can only be ensured if
randomization is concealed

Concealment
of allocation (Concealment of allocation is commonly misunderstood
to mean blinding. Blinding and concealment are quite
different features of clinical trials. It would probably be
clearer if concealment of allocation was called
concealment of recruitment) .
This means that the researcher is not aware, at the time
a decision is made about eligibility of a person to
participate in the trial, if that per- son would
subsequently be randomized to the treatment or control
group.
 Concealment is important because, even though most
trials specify inclusion and exclusion criteria that
determine who is and who is not eligible to participate
in the trial, there is sometimes uncertainty about
whether a particular patient satisfies those criteria, and
Concealment often the researcher responsible for entering new
patients into the trial has some latitude in such
of allocation decisions.

It could seriously bias the trial’s findings if the

researcher’s decision about who was and was not
entered into the trial was influenced by knowledge of
which group patients would subsequently be assigned
to.
2. Was there complete or near complete follow-up?

• Doing clinical trials is hard and often mundane work. One of the difficulties is
ensuring that the trial protocol is adhered to. And one of the hardest parts of the
trial protocol to adhere to is the planned measurement of outcomes (‘follow-up’).
Was there complete or near complete
follow-up?

• Most clinical trials involve interventions that are implemented over days
or weeks or months. Usually outcomes are assessed at the end of the
intervention, and they are often also assessed at one or several times after
the intervention has ceased. Trials of chronic conditions may assess out-
comes several years after the intervention period has ceased.
 Problems with Follow-up

• A problem that arises in most trials is that it is not always possible to obtain
outcome measures as planned. Occasionally subjects die. Others become too sick
to measure, or they move out of town, or go on long holidays. Some may lose
interest in participating in the study or simply be too busy to attend for follow-up
appointments
Loss to follow-up/ Dropouts

• For these and a myriad of other reasons it may be impossible for the researchers
to obtain outcome measures from all subjects as planned, no matter how hard
the researchers try to obtain follow-up measures from all patients. This
phenomenon of real-life clinical trials is termed ‘loss to follow-up’. Subjects lost to
follow-up are sometimes called ‘dropouts
Random Vs Non Random dropouts

• Loss to follow-up would be of little concern if it occurred at random. But in

practice loss to follow-up may be non-random, and this can produce bias. Bias
occurs when dropouts from one group differ systematically, in terms of their
outcomes, from dropouts in the other group

• When this occurs, differences between groups are no longer attributable just to
the intervention and chance
Hypothetical Example
• Imagine a hypothetical trial of treatment for cervical headache. The trial compares the
effect of six sessions of manual therapy to a no-intervention control condition, and
outcomes in both groups are assessed 2 weeks after randomization.
• Some subjects in the control group may experience little resolution of their symptoms.
Understandably, these subjects may become dissatisfied with participation in the trial and
may be reluctant to return for outcome assessment after not having received any
intervention.
• The consequence is that there may be a tendency for those subjects in the control group
with the worst outcomes to be lost to follow-up, more so than in the treated group. In that
case, estimates of the effects of intervention are likely to be biased and the treatment will
appear less effective than it really is.
Acceptable range of Dropouts

• How much loss to follow-up is required to seriously threaten the validity of a

study’s findings? Many statisticians would not be seriously concerned with
dropouts of as much as 10% of the sample. On the other hand, if more than 20%
of the sample was lost to follow-up there would be grounds for concern about
the possibility of serious bias. A rough rule of thumb might be that, if greater
than 15% of the sample is lost to follow-up then the findings of the trial could be
considered to be in doubt.
Important consideration in critical appraisal

• Studies which do not provide data on loss to follow-up and which

do not explicitly state that there was no loss to follow-up should be

considered potentially biased.

protocol violation

• Protocol violations occur when the trial is not carried out as planned. In trials of physiotherapy
interventions, the most common protocol violation is the failure of subjects to receive the intended
intervention.

• For example, subjects in a trial of exercise may be allocated to an exercise group but may fail to do
their exercises, or fail to exercise according to the protocol (this is sometimes called ‘non-
compliance’ or ‘non-adherence’), or subjects allocated to the control condition may take up exercise.

• Protocol violations are undesirable, but usually some degree of protocol violations cannot be
avoided. Usually they present less of a problem than loss to follow-up.
What to do in case of Protocol violation ?
• The most satisfactory solution is the least obvious one. It involves ignoring the protocol violations and
analysing the data of all subjects in the groups to which they were allocated. This is called ‘analysis by
intention to treat’

• analysis by intention to treat preserves the benefits of randomization

• it maintains the comparability of groups. Also, from a pragmatic point of view, analysis by intention to
treat provides the most meaningful estimates of effects of intervention. This is because, pragmatically
speaking, interventions can only be effective if patients comply
3. Was there blinding to allocation of patients and assessors?

• There is reason to prefer that, in clinical trials, subjects are unaware of whether
they received the intervention or control condition. This is called blinding of
subjects.
• Blinding of subjects is considered important because it provides a means of
controlling for placebo effects.

• Blinding of subjects ensures that estimates of the effects of intervention (the

difference between outcomes of treated and control groups) cannot be due to
placebo effects.
How is it possible to blind patients to allocation?
How can subjects not know if they received the intervention or control?

• The general approach involves giving a ‘sham’ intervention to the control group.

• Sham interventions are those that look, feel, sound, smell and taste like the
intervention but could not effect the presumed mechanism of the intervention.
Assessor Masking

• Wherever possible, assessors (the people who measure outcomes

in clinical trials) should be unaware, at the time they take each measurement of
outcome, whether the measurement is being made on someone who received
the intervention or control condition.
SYSTEMATIC REVIEWS OF RANDOMIZED TRIALS

• If a systematic review is to produce valid conclusions it must identify most of the relevant studies
that exist and produce a balanced synthesis of their findings. To determine if this goal has been
achieved, readers can ask three questions.
1. Was it clear which trials were to be reviewed?
2. Were most relevant studies reviewed?
3. Was the quality of the reviewed studies taken into account?
Was it clear which trials were to be reviewed?

• When we read systematic reviews we need to be satisfied that the reviewer has not
selectively reviewed those trials which support his or her own point of view

• To reduce the possibility of selective reviewing, reviewers should clearly define the scope
of the review prior to undertaking a search for relevant trials. The best way to do this is
to clearly describe criteria that are used to decide what sorts of trials will be included in
the review, and perhaps also which trials will not. The inclusion and exclusion criteria
usually refer to the population, interventions and outcomes of interest.
Was it clear which trials were to be
reviewed?
• Systematic reviews which specify clear inclusion and exclusion
criteria provide stronger evidence of effects of therapy than
those that do not.
Were most relevant studies reviewed?

• Locating all relevant trials is not an easy task

• Well-conducted reviews identify most trials relevant to the review question.

• Randomized trials in physiotherapy are indexed across a range of partially overlapping major
medical literature databases such as Medline, Embase, CINAHL, AMED, and PsycINFO. The
Cochrane Collaboration’s Register of Clinical Trials and the Centre for Evidence-Based Physio-
therapy’s PEDro database attempt to provide more complete indexes of the clinical trial literature,
but they rely on other databases to locate trials.
Were most relevant studies reviewed?

• Some trials are not indexed on any databases, or are so poorly indexed that they
are unlikely ever to be found. they may be published in obscure journals, or they
may not have been published at all.
Publication/ Language Bias

• Systematic reviews which search only for published trials are said to be exposed
to ‘publication bias’, and systematic reviews which search only for trials reported
in English are said to be exposed to ‘language bias’.
Was the quality of the reviewed studies taken into account?

• Many randomized trials are poorly designed and provide potentially seriously
biased estimates of the effects of intervention. Consequently, if a systematic
review is to obtain an unbiased estimate of the effects of intervention, it must
ignore low quality studies.

• The simplest way to incorporate quality assessments into the findings of a

systematic review is to list minimum quality criteria for trials.
Was the quality of the reviewed studies
taken into account?
• The most popular methods used to assess trial quality in systematic reviews of
physiotherapy are the Maastricht scale (Verhagen et al 1998b), the Cochrane
Back Review Group criteria (van Tulder et al 2003), the Jadad scale (1996) and the
PEDro scale (Maher et al 2003). Two of these, the Maastricht scale and the PEDro
scale, generate a quality score (that is, they are scales), and the other two do not
(they are checklists).
Summary
Critical Appraisal of Evidence about
Experience
• Qualitative research uses methods that are substantively different from most
quantitative research. The methods differ with regard to
• sampling techniques,
• data collection methods and
• data analysis.

Consequently, the criteria used to appraise qualitative research must differ from
those used to appraise quantitative research.
Critical Appraisal of Evidence about
Experience
• When critically appraising the methodological quality of qualitative research, you
need to ask questions that focus on other elements and issues than those that
are relevant to research which includes numbers and graphs.

• Appraisal should focus on the trustworthiness, credibility and dependability of

the study’s findings – the qualitative parallels of validity and reliability
CRITICAL APPRAISAL OF EVIDENCE ABOUT EXPERIENCES

• When readers look to reports of qualitative research to answer clinical questions

about experiences, it is suggested they routinely consider the following three
issues.

1. Was the sampling strategy appropriate?

2. Was the data collection sufficient to cover the phenomena?
3. Were the data analysed in a rigorous way?
1. Was the sampling strategy appropriate?

• Why was this sample selected?

• How was the sample selected?

• Were the subjects’ characteristics defined?

In qualitative research we are not interested in an ‘on average’ view of a

population. We want to gain an in-depth understanding of the experience of
particular individuals or groups.
1. Was the sampling strategy appropriate?
The sample in qualitative research is often made up of

• individual people,
• situations,
• social settings,
• social interactions or documents.

The sample is usually strategically selected to contain subjects with relevant roles,
perspectives or experiences.
purposive sampling

• The conscious selection of a small number of individuals meeting particular

criteria – a process called purposive sampling

snowball sampling

• Sometimes sampling occurs in an opportunistic way: one person leads the

researcher to another person, and that person to one more, and so on. This is

called snowball sampling

1. Was the sampling strategy appropriate?

• The author should explain and justify

• why the participants in the study were the most appropriate to provide access to
the type of knowledge sought by the study.

• If there have been any problems with recruitment (for example, if there were
many people that were invited to participate but chose not to take part), this
should be reported.
2. Was the data collection sufficient to cover the phenomena?

• Was the method used to collect data relevant?

• Were the data detailed enough to interpret what was being researched?
2. Was the data collection sufficient to
cover the phenomena?
• A range of very different methods is used to collect data in qualitative research. These vary from,
for example,

• participant observations,

• in-depth interviews,

• focus groups,

• document analysis.

The data collection method should be relevant and address the questions raised, and should be
justified in the research report.
Was the data collection sufficient to cover
the phenomena?
• Data collection should be comprehensive enough in both breadth (type of
observations) and depth (extent of each type of observation) to generate and
support the interpretations.

• That means that as much data as possible should be collected.

Ethical consideration
• Another important question to ask about data collection is whether ethical issues
have been taken into consideration.
• The ethics of a study do not have a direct bearing on the study’s validity but may,
nonetheless, influence a reader’s willingness to read and use the findings of the
study.
• In qualitative research, peoples’ feelings and deeper thoughts are revealed and it
is therefore important that issues around informed consent and confidentiality
are clarified.
Were the data analysed in a rigorous way?

• Was the analytical path described?

• Was it clear how the researchers derived categories or themes from the
data, and how they arrived at the conclusion?

• Did the researchers reflect on their roles in analysing the data?

Were the data analysed in a rigorous way?

• The process of analysis in qualitative research should be rigorous.

• This is a challenging, complex and time-consuming job.

• The aim of this process is often to make sense of an enormous amount of text,
tape recordings or video materials by reducing, summarizing and interpreting the
data.
Triangulation

• There are several features that can strengthen a reader’s trust in the findings of a
study.

• One is the use by the researchers of more than one source for information when
studying the phenomena, for example the use of both observation and
interviews. This is often called triangulation.
Triangulation

• Triangulation might involve the use of more than one method, more than
one researcher or analyst, or more than one theory.

• The use of more than one investigator to collect and analyse the raw data
(multiple coders) also strengthens the study.
CRITICAL APPRAISAL OF EVIDENCE ABOUT PROGNOSIS

• INDIVIDUAL STUDIES OF PROGNOSIS

1. Was there representative sampling from a well-defined population?

2. Was there an inception cohort?
3. Was there complete or near-complete follow-up?
1. Was there representative sampling from a well-defined population?

• When we read studies of prognosis we first need to know which population the
study is seeking to provide a prognosis for (the ‘target population’).

• The target population is defined by the criteria used to determine who was
eligible to participate in the study. Most studies of prognosis describe a list of
inclusion and exclusion criteria that clearly identify the target population.
1. Was there representative sampling from
a well-defined population?
• When you read studies looking for information about
prognosis, start by looking to see whether the study
recruited ‘all’ patients or ‘consecutive cases’. If it did not, the
study may provide biased estimates of the true prognosis.
2. Was there an inception cohort?

• survivor cohorts of life-threatening diseases might generate unrealistically good prognoses. Either
way, survivor cohorts potentially provide biased estimates of prognosis.

• The solution is to recruit subjects at a uniform (usually early) point in the course of the disease.39
Studies which recruit subjects in this way are said to recruit ‘inception cohorts’ because subjects
were identified as closely as possible to the inception of the condition. The advantage of inception
cohorts is that they are not exposed to the biases inherent in studies of survivor cohorts.

• Studies that recruit inception cohorts may provide less biased estimates of prognosis than
studies that recruit survivor cohorts.
3. Was there complete or near-complete follow-up?

• Like clinical trials of effects of therapy, prognostic studies can be biased by loss to
follow-up. Bias occurs if those lost to follow-up have, on average, different
outcomes to those who were followed up.

• as a rough rule of thumb, the study is unlikely to be seriously biased by loss to

follow-up if follow-up is at least 85%.