Critical Appraisal for Therapy Articles

Title of the appraised study:

Are the results of the trial valid? (Internal Validity)

What question did the study ask?
Patients -

Intervention -

Comparison -

Outcome(s) -

1a. R- Was the assignment of patients to treatments randomised?

What is best? Where do I find the information?
Centralised computer randomisation is ideal and The Methods should tell you how patients were
often used in multi-centred trials. Smaller trials may allocated to groups and whether or not randomisation
use an independent person (e.g., the hospital was concealed.
pharmacy) to “police” the randomization.
This paper: Yes  No  Unclear 
Comment:

1b. R- Were the groups similar at the start of the trial?

What is best? Where do I find the information?
If the randomisation process worked (that is, The Results should have a table of "Baseline
achieved comparable groups) the groups should be Characteristics" comparing the randomized groups
similar. The more similar the groups the better it is. on a number of variables that could affect the
There should be some indication of whether outcome (i.e. age, risk factors etc.). If not, there may
differences between groups are statistically be a description of group similarity in the first
significant (i.e. p values). paragraphs of the Results section.
This paper: Yes  No  Unclear 
Comment:

2a. A – Aside from the allocated treatment, were groups treated equally?
What is best? Where do I find the information?
Apart from the intervention the patients in the Look in the Methods section for the follow-up
different groups should be treated the same, e.g., schedule, and permitted additional treatments, etc.
additional treatments or tests. and in Results for actual use.
This paper: Yes  No  Unclear 
Comment:

2b. A – Were all patients who entered the trial accounted for? – and were they analysed in
the groups to which they were randomised?
What is best? Where do I find the information?
Losses to follow-up should be minimal – preferably The Results section should say how many patients
less than 20%. However, if few patients have the were randomised (e.g., Baseline Characteristics
outcome of interest, then even small losses to table) and how many patients were actually included
follow-up can bias the results. Patients should also in the analysis. You will need to read the results
be analysed in the groups to which they were section to clarify the number and reason for losses to
randomised – ‘intention-to-treat analysis’. follow-up.
This paper: Yes  No  Unclear 
Comment:

3. M - Were measures objective or were the patients and clinicians kept “blind” to which
treatment was being received?
What is best? Where do I find the information?
It is ideal if the study is ‘double-blinded’ – that is, First, look in the Methods section to see if there is
both patients and investigators are unaware of some mention of masking of treatments, e.g.,
treatment allocation. If the outcome is objective placebos with the same appearance or sham
(e.g., death) then blinding is less critical. If the therapy. Second, the Methods section should
outcome is subjective (e.g., symptoms or function) describe how the outcome was assessed and
then blinding of the outcome assessor is critical. whether the assessor/s were aware of the patients'
treatment.
This paper: Yes  No  Unclear 
Comment:

University of Oxford, 2005 1

What were the results?
1. How large was the treatment effect?
Most often results are presented as dichotomous outcomes (yes or no outcomes that happen or don't
happen) and can include such outcomes as cancer recurrence, myocardial infarction and death. Consider a
study in which 15% (0.15) of the control group died and 10% (0.10) of the treatment group died after 2 years
of treatment. The results can be expressed in many ways as shown below.

Diarrhea in 5 days (+) Diarrhea in 5 days (-)

Synbio 14 65 79
Placebo 30 47 77
44 112 156

What is the measure? What does it mean? Results on the appraised study:
Relative Risk (RR) = The relative risk tells us how many times 0.45
risk of the outcome in more likely it is that an event will occur in
the treatment group / the treatment group relative to the control
risk of the outcome in group. An RR of 1 means that there is no
the control group. difference between the two groups thus,
the treatment had no effect. An RR < 1
means that the treatment decreases the
risk of the outcome. An RR > 1 means that
the treatment increased the risk of the
outcome.
In our example, the RR Since the RR < 1, the treatment decreases Group B.lactis B94 plus a prebiotic
= 0.10/0.15 = 0.67 the risk of death. inulin (synbio) had 55% lower risk
of still having diarrhea on 5th day
compared to placebo.

Absolute Risk The absolute risk reduction tells us the CER = 0.39
Reduction (ARR) = risk absolute difference in the rates of events
EER = 0.18
of the outcome in the between the two groups and gives an
control group - risk of indication of the baseline risk and ARR = 0.21
the outcome in the treatment effect. An ARR of 0 means that
treatment group. This is there is no difference between the two
also known as the groups thus, the treatment had no effect.
absolute risk
difference.
In our example, the ARR The absolute benefit of treatment is a 5% The absolute benefit of treatment
= 0.15 - 0.10 = 0.05 or reduction in the death rate. is a 21% reduction of having
5% diarrhea ≥5 days
Relative Risk The relative risk reduction is the RRR = 0.55
Reduction (RRR) = complement of the RR and is probably the
absolute risk reduction / most commonly reported measure of
risk of the outcome in treatment effects. It tells us the reduction in
the control group. An the rate of the outcome in the treatment
alternative way to group relative to that in the control group.
calculate the RRR is to
subtract the RR from 1
(e.g. RRR = 1 - RR)
In our example, the RRR The treatment reduced the risk of death by The treatment reduced the risk of
= 0.05/0.15 = 0.33 or 33% relative to that occurring in the control having diarrhea ≥5 days by 55%
33% group. relative to that occurring in the
Or RRR = control group.
1 - 0.67 = 0.33 or 33%
Number Needed to The number needed to treat represents the NNT = 4.71
Treat (NNT) = inverse of number of patients we need to treat with
the ARR and is the experimental therapy in order to
calculated as 1 / ARR. prevent 1 bad outcome and incorporates
the duration of treatment. Clinical
significance can be determined to some
extent by looking at the NNTs, but also by
weighing the NNTs against any harms or
adverse effects (NNHs) of therapy.
In our example, the NNT We would need to treat 20 people for 2 We would need to treat 5 people in
= 1/ 0.05 = 20 years in order to prevent 1 death. order to prevent 1 occurrence of
diarrhea ≥5 days.
2. How precise was the estimate of the treatment effect?

University of Oxford, 2005 2

 The true risk of the outcome in the population is not known and the
best we can do is estimate the true risk based on the sample of
patients in the trial. This estimate is called the point estimate. We can
gauge how close this estimate is to the true value by looking at the
confidence intervals (CI) for each estimate. If the confidence interval is
fairly narrow, then we can be confident that our point estimate is a
precise reflection of the population value. The confidence interval also
provides us with information about the statistical significance of the
result. If the value corresponding to no effect falls outside the 95%
confidence interval, then the result is statistically significant at the 0.05
level. If the confidence interval includes the value corresponding to no
effect, then the results are not statistically significant.

University of Oxford, 2005 3

Will the results help me in caring for my patient? (External Validity/Applicability)
The questions that you should ask before you decide to apply the results of the study to your patient are:
 Is my patient so different to those in the study that the results cannot apply?

 Is the treatment feasible in my setting?

 Will the potential benefits of treatment outweigh the potential harms of treatment for my patient?

University of Oxford, 2005 4