Meta-analyses and indirect comparisons

Alain Duhamel, Cristian Preda

CHRU Lille and Inria Lille

Université de Lille

alain.duhamel@univ-lille2.fr
cristian.preda@polytech-lille.fr

154-th ICB Seminar on Statistics and Clinical Practice

Warsaw,11-13 May, 2017
A>B?
Question : Is the treatment A better than the treatment B ?

What is "the best available evidence" ?

Study Number of subjects Mortality (A) Mortality (B) p-value

S1 56 5.9% 5.2 % NS
S2 100 6.0% 2.0 % NS
S3 395 2.5% 6.5 % NS
S4 52 4.3% 17.2 % NS
S5 103 4.2% 3.5 % NS
S6 301 1.9% 7.3 % < 0.05

Two possible conclusions :

1. no evidence for "A > B"
but maybe we have a problem of statistical power (small sizes)
2. evidence for "A > B"
but there is a large sample size study (study S3) saying the contrary.
Meta-analysis
The two conclusions are possible because of an arbitrary choice of
studies !

Meta-analysis : a statistical technique aiming to :

I combine the results of two or more studies,
I estimate an average or a common effect.
Meta-analysis

Functions of meta-analysis :

I Quantify treatments effects and their uncertainty,

I Increase statistical power and precision (to detect an effect),

I Explore differences between studies (identify heterogeneity),

I Reduce the subjectivity of study comparison by using systematic and

explicit comparison procedure,

I Generate new hypotheses (e.g. calculate the sample size for future
studies).
Meta-analysis and quality of evidence
The evidence hierarchy :

The systematic reviews and the meta-analyses of several randomized

trials has become the gold standard for judging whether a treatment does
more good than harm.
Meta-analysis and systematic review
Remark :
I A systematic review answers a defined research question by
collecting and summarising all empirical evidence that fits
pre-specified eligibility criteria.
I A meta-analysis is the use of statistical methods to summarise the
results of these studies.

Sometime, it is considered that

meta-analysis = systematic review + statistical analysis

Quality of a meta-analysis

A meta-analysis is only as goof as the studies in it !

I if included studies are biased the result of the meta-analysis will also
be incorrect.
I if reporting biases are present, unrepresentative set of studies may
give misleading result.
Steps of meta-analysis :
1. Define the research question and specific hypotheses.
Example : High-dosage vitamin E supplementation say increase
all-cause mortality ?
2. Defining criteria for including and excluding studies. PICOS criteria :
Patients, Intervention, Comparisons, Outcomes, Study design.
Example : RCT studies with JADAD score > 4, vitamin E intake,
man and no-pregnant women, etc).

Cochran Collaboration proposes almost similar criteria.

Steps of meta-analysis :
3. Locate research studies (PubMed, Embase, Cochran CDRS, Clinical
Trials, reference papers and abstracts).
Example : Edgar R. Miller et al. High-Dosage Vitamin E Supplementation May Increase
All-Cause Mortality. Ann Intern Med.2005 ;142 :37-46. "We performed a MEDLINE search by
using the Medical Subject Heading (MeSH) terms vitamin E, antioxidant vitamins, alpha

tocopherol, tocopherol, and clinical trials. The search period was 1966 through August 2004. We
complemented the MEDLINE search by searching the Cochrane database of randomized,
controlled trials ; reviewing the reference lists from original research, review articles, and previous
meta-analyses ; and reviewing the files of the investigators" ..."Three investigators independently
abstracted the articles. They resolved disagreements by consensus".

4. Determine which studies are eligible for inclusion.

5. Classify and code important characteristics of the studies (sample
sizes, outcome definition, treatment brand and dose, etc).
6. Select end translate the results of each study using a common scale.
7. Aggregate findings across studies by generating pooled estimates of
the effect size.
8. Evaluate the statistical homogeneity of the pooled studies.
9. Perform sensibility analysis.
Bias publication :
Studies with significant results are more likely
- to be published,
- to be written in English,
- to be cited by others,
- to produce multiple publications.

Including only published studies can introduce publication bias (often

ignored by the systematic reviews !)
Methods for detecting bias publications (e.g., Funnel plots and Egger
test).
Meta-analysis : the flow-chart
Meta-analysis : estimation of the size effect

Outcome type and the measured effect :

- scalar :
I mean difference,
I mean difference with respect to a baseline,
I relative mean difference.
- categorical
I frequence difference,
I relative risque (RR)
I odds ratio (OR)
- survival
I the hazard rate
Meta-analysis : estimation of the size effect

The tool : (Mixed) linear model :

θ = the true effect

Let Ti the observed effect for the i − th study.
I The fixed effects linear model :

Ti = θ + i , i ∼ N (0, σi2 )
I The random effects linear model :

Ti = θ + αi + i , , αi ∼ N (0, σs 2), i ∼ N (0, σi2 )

Meta-analysis : estimation of the size effect
Fixed effects model : ML estimator,
n
wT 1
Pni i ,
X
θ̂ = wi =
i=1 i=1 wi σi2

1
var (θ̂) = Pn
i=1 wi
Random effects model :
1
wi =
σs2 + σi2
The Cochran Q statistic,
n Pn 2
i=1 wi Ti
X
Q= wi Ti2 − P n
i=1 i=1 wi

is distributed as a χ2 (n − 1) if homogeneity between studies.

Meta-analysis : random or fixed models

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

(PRISMA) : an evidence-based minimum set of items for reporting in
systematic reviews and meta-analyses.
Meta-analysis : the forest plot
Example (Miller et al., Ann Intern Med. 2005 Jan 4 ;142(1) :37-46).
Meta-analysis : high-dosage vitamin E supplementation may increase
all-cause mortality.
Meta-analysis : heterogenity

Heterogeneity = variation between the studies’ results.

I Causes of heterogeneity : differences between studies wrt :
- subjects (inclusion and exclusion criteria, age, etc),
- intervention (dose, duration, type, etc),
- outcome (type, scale, cut-off, etc)
- quality (randomization, blinding, etc)

I How to look for heterogeneity ?

- visually : On the forest plot do the confidence intervals overlap
with each other and the overall effect ?
- statistically : Chi-square test for heterogeneity (Cochran Q
test)
Meta-analysis : the funnel plot and publication
bias
Meta-analysis : limitations

I A meta-analyse reflects only what is published and searchable !

I It is focussed on mean effects and differences between studies. But

what really matters is the effects on individuals !.
Indirect comparisons and meta-analysis
Indirect comparison
Aim : compare treatment A to treatment B

Direct comparison : treatment A is compared to treatment B

Indirect comparison : there exists a treatment C and several studies

comparing A to C and B to C are available. None study compares A to B !
Indirect comparison

Indirect comparison has multiple causes :

I sometime one can not impose the comparison to the reference (e.g.,
for symptomatic treatments, comparison with Placebo is acceptable !
I To impose comparison to the reference could lead to complications :

T1 > P, next T2 > T1 , next T3 vsT2 : NS

How to proceed in this case : abandon of T3 ? Maybe it is better

alternative to T1 (no well tolerated) ! We need to compare T1 to T3 .

Since A and B have a common reference (C), how to compare them ?

Indirect comparison : Naive methods

RR(A, B) = 0.15/0.1 = 1.5, so B is better than A ! But we forget the

baseline ! (not the same population)

RR(A, P) = 0.5, RR(B, P) = 0.6. It is A better than B ?

- 95% confidence interval for RR(A/P) = (0,25 ; 0,75)

- 95% confidence interval for RR(B/P) = (0,55 ; 0,65)

A could have a smaller effect (25%) than B (35%) !
Indirect comparison : general methodology

Necessary conditions :

I several studies comparing A versus C and B versus C are available.

I the selected studies fulfil the requirements to be included into a
meta-analysis.
I the meta-analysis of A versus C and the meta-analysis of B versus
C both yield to homogeneity.
I similarity principle : clinical and methodological similarity between
A/C and B/C (patients, outcome, models, bias, etc)

Then, an overall effect θAB can be estimated, e.g.

RR(A/P)
RR(A/B) =
RR(B/P)
Indirect comparison : general methodology
Bucher method (JCE 1997)

Let define

θ̂AC = the pooled effect of the meta-analysis for A versus C

and

θ̂BC = the pooled effect of the meta-analysis for B versus C

Then, the pooled effect of A versus B, θ̂A∼B is given by the Chasles

relation,

θ̂A∼B = θ̂AC − θ̂BC

var (θ̂A∼B ) = var (θ̂AC ) + var (θ̂BC )

θ̂A∼B − θAB
Z=q ∼ N (0, 1)
var (θ̂A∼B )
Indirect comparison : mixed treatment comparison
(MTC)
Mixed treatment comparisons (MTC) : in addition to indirect
comparisons, there exists direct comparisons for A and B.

Let denote by θ̂AB the pooled effect from the meta-analysis comparing
directly A and B.

Then, the consistency condition requires that θ̂A∼B and θ̂AB are both
estimators of the same quantity ΘAB :

θ̂A∼B − θ̂AB
Z=q ∼ N (0, 1)
var (θ̂A∼B ) + var (θ̂AB )
If the consistency condition holds then the pooled effect from the direct
and the indirect comparisons is given by

w1 θ̂A∼B + w2 θ̂AB 1
Θ̂AB = , var (Θ̂AB ) = ,
w1 + w2 w1 + w2
where w1 = 1/var (θ̂A∼B ) and w2 = 1/var (θ̂AB ).
Indirect comparison :
Several paths for indirect comparisons :

To compare D and A there are two common references : C and Placebo.

References and software
References and software

I Randomized trials : PRISMA (Preferred Reporting Items for

Systématic Revieews and Meta-Analysis) recommendations :
(Alessandro Liberati et al. The PRISMA statement for reporting
systematic reviews and meta-analyses of studies that evaluate health
care interventions : explanation and elaboration. Journal of Clinical
Epidemiology 62 (2009) e1- e34)
See also : http ://www.prisma-statement.org
References and software

I Observational studies : MOOSE Recommandations (Meta-analysis

Of Observational Studies in Epidemiology)
(Stroup DF et al. Meta-analysis of observational studies in
epidemiology : a proposal for reporting. Meta-analysis Of
Observational Studies in Epidemiology (MOOSE) group.JAMA.
2000 Apr 19 ;283(15) :2008-12)

I Cochrane Collaboration :

http ://www.cochrane.org
References and software
I Cochran RevMan software :

http ://tech.cochrane.org/revman

I R software : packages meta, metafor, rmeta