Training on

Systematic Review and Meta-analysis

For Debre Markos University academic staffs

BY

Gedefaw Diress (MPH, Ass’t professor of Epidemiology)

Data Analysis and Meta-analysis


Meta-analysis
 It is a systematic procedure for summarizing and pooling
the results from 2 or more research studies.

 We combine the results of similar individual studies, to give

a cumulative result, which may then differ from the results
of the individual studies.
Aim of meta-analysis

 To increase precision of overall estimate of “effect”.

 To increase strength of correlation or association between

exposure and outcome

 To investigate reasons for differences in risk estimates or

discover patterns of risk amongst studies.

 To discover patterns of risk amongst studies

Aim of meta-analysis

 However, they also have the potential to mislead seriously,

particularly if specific study designs, within-study biases,
variation across studies, and reporting biases are not
carefully considered.
When is meta-analysis useful?

1. When there is variation in the effect size

 if each study reports a difference in treatment effects, such as

the percentage of reduction in the incidence of a condition.
When is meta-analysis useful?

2. Many studies with small sample size

 Often a study does not carry enough weight to make it statistically

significant because of its sample size.

 If weable to combine the multiple, good quality, and similar small

studies, this can increase the power and possibly provide a
meaningful result.
When is meta-analysis useful?
3. When we need a definitive conclusion on the effectiveness
of an intervention

 We should not use the results of one single study to provide us with
a definitive conclusion on the effectiveness of an intervention, but
rather pool the results of multiple similar studies to confirm the
effectiveness.
When meta-analysis is appropriate

 Meta analysis will be appropriate when we:

 Have the same population

 Use the same intervention administered in the same way.

 Measure the same outcomes

 Homogeneity
 studies are sufficiently similar to estimate an average effect.
When meta-analysis is appropriate
 Studies to be included in meta-analysis should be similar to each other
so that generalisation of results is valid.

 This is referred as homogeneity.

 The four main criteria that must be considered are:

 patient population
 outcome
 Intervention
 control
 Possible statistics
I. Prevalence or proportion
II. Mean
III. Mean difference
IV. Risk difference
V. Relative risk
VI. Odds ratio
VII. Hazard ratio
Confidence Intervals for effect estimate
 Confidence intervals are an indication of how precise the
findings are
 Sample size greatly impacts the CI
 thelarger the sample size the smaller the CI, the greater the
power and confidence of the estimate
CIs indicate:
 When calculated for OR, the CI provides the upper and
lower limit of the odds that a treatment may or may not
work
 If the odds ratio is 1, odds are even and therefore, not
significantly different
 recall the odds of having a boy
 Statistical Information Needed to Calculate Effect Sizes
14

 Total population(N) for each treatment and

control/comparison group on each outcome and at each
data collection point

 To calculate effect sizes for continuous measures (scales,

numeric data), we will need
 Means
For each treatment and
 standard deviations
control or comparison
 valid N’s group

Gedefaw D. 5/31/2023
 Statistical Information Needed to Calculate Effect Sizes
15

 To calculate effect sizes for continuous measures (scales,

numeric data), we will need
 Means
 standard deviations
 valid N’s
 Record all information on statistics used to test for
differences between groups (e.g., t-test, F statistic, p
values), especially if any of the raw data (means and
standard deviations) are not available.

Gedefaw D. 5/31/2023
 Statistical Information Needed to Calculate Effect Sizes
16

 Effect sizes for dichotomous data require information on

the
 number of treatment cases that experienced an event

 number of control cases that experienced the event

 total valid N’s for each group.

Gedefaw D. 5/31/2023
 Effect size and pooling methods
17

 Pooling data across studies is the purpose of meta-

analysis
 An effect size is a measure of the strength (magnitude)
and direction of a relationship between variables

 Any metric can be used as an ES as long as it is takes into

account the magnitude and direction of a relationship

 Most ES metrics fall into three main categories, related to

proportions, means, and correlation coefficients
Gedefaw D. 5/31/2023
 Effect size and pooling methods
18

 Different ES measures are used for dichotomous and

continuous data.

 Although each individual can have only one value on

these variables, grouped data can be expressed in
proportions or rates.

 For example, although pregnancy is a dichotomous

variable, we can calculate the proportion of the women in
a group who become pregnant and compare pregnancy
rates in different groups.
Gedefaw D. 5/31/2023
 Effect size and pooling methods
19

 Continuous variables can take on a range of values that

can be expressed on a numeric scale.

 Examples: number of pregnancies, number or length of

hospitalizations, days in out-of-home placement, and
years of education

Gedefaw D. 5/31/2023
 Effect Sizes for Dichotomous Data
20

Common ES measure for dichotomous data are

1. Odds ratio (OR)

 The OR is a comparison of two odds; that is, it is the odds that
something will happen in one group compared with the odds
that it will happen in the other.
2. Risk ratio (or relative risk, RR)
 compares the chance of an event in one group with the chance
of that event in another group

Gedefaw D. 5/31/2023
21

Gedefaw D. 5/31/2023
 OR vs RR
22

 The difference between OR and RR is small when events

are rare and larger when events are more common
 An OR of 2:1 versus 1:2 equals 4, while the RR for the
same data is 0.67/0.33= 2.
 The risk is twice as likely, and the odds are four times
greater.
 Because ORs produce more extreme values than RRs, this
misinterpretation leads readers to overestimate large
effects and underestimate small effects

Gedefaw D. 5/31/2023
 Cont..
23

 Many meta-analysts use the natural logarithm of the OR, or

log odds ratio, because it has better statistical properties.

 The log OR is distributed around 0 (instead of 1) and has an

approximately normal distribution (standard deviation
[SD]=1.83).

 Positive values represent an increase in the odds; negative

values represent a decrease in the odds.

 Log OR can be converted back to OR for ease of

interpretation.

Gedefaw D. 5/31/2023
 Cont..
24

 If confidence limits are given instead of a standard error

(SE), extra computation is required.

 Consider a relative risk estimate RR with a given 95%

LBRR and UBRR.

 Standard error is given by SE = (ln UBRR - In LBRR)/3.92,

where 3.92 = 2 X1.96 is twice the normal percentile for
95% limits

Gedefaw D. 5/31/2023
 Effect Sizes for Continuous Data
25

 Mean difference
 The average of the treatment group minus the average of the control
group

 Standardized mean difference (SMD)

 Useful when scores are reported in different ways or different scales
are used to assess the same construct
 The mean difference divided by the pooled SD of the two groups

 Correlation Effect Size

 expresses the strength and direction of an association between two
continuous variables

Gedefaw D. 5/31/2023
 Other Effect Size Metrics
26

 Less common ES metrics include

 hazard rates for time-to-event data,
 ordinary least squares (OLS) regression coefficients, results
of factor analysis, and data from receiver operating
characteristics (ROC) curves.

Gedefaw D. 5/31/2023
 Corrections for Error and Bias
27

 Two statistical sources of influence on the ES:

 Small sample bias and
 Unreliable outcome measures.

 ESs from small samples can be unduly influenced by a

single outlier.

 The presence of small-sample bias may be detected by

examining the Forest plot closely or by creating funnel
plots

Gedefaw D. 5/31/2023
 Analyses on Meta-data
28

 To work on aggregate data, first we can prepare

data on Excel or SPSS and transport to software
that could run Meta analysis

Gedefaw D. 5/31/2023
 Installing STATA packages for Meta Analysis
29

It is needed if the Stata software is less than Version 16

We can install STATA packages for Meta data in two ways:

Option I: Installing each package separately:

Example: ssc install metan ………….. for main meta analysis

 ssc install metareg........... for meta regression
 ssc install metaninf ......... for sensitivity analysis
 ssc install metafunnel …….. for funnel plot
 ssc install metabias ……… for Egger’s & Beg’s test
 ssc install metatrim …….. Trim and fill analysis
Gedefaw D. 5/31/2023
 Installing STATA packages for Meta Analysis
30

Option II: Installing packages from within Stata online

 To install from within Stata online, click on:

Help SJ and user written commands STB then do

the following:

Gedefaw D. 5/31/2023
 Installing STATA packages for Meta Analysis
31

If our Stata is Stata 16 and above no need of installing the

package

Gedefaw D. 5/31/2023
 Exercise
32

Research question

Among trafficked women (population), were sex

trafficked women (exposed) at a higher risk of
developing HIV/AIDS (outcome) as compared to those
who were trafficked for housemaid purpose (comparison)?

Gedefaw D. 5/31/2023
 Exercise
33

Research question

Among trafficked women (population), were sex

trafficked women (exposed) at a higher risk of
developing HIV/AIDS (outcome) as compared to those
who were trafficked for housemaid purpose (comparison)?

Gedefaw D. 5/31/2023
 Exercise
34

Gedefaw D. 5/31/2023
 Analyses on Meta-data
35

Gedefaw D. 5/31/2023
 Analyses on Meta-data
36

Gedefaw D. 5/31/2023
 Analyses on Meta-data
37

 Estimate the magnitude of true heterogeneity and

test it (Qstatistic), visually check it using forest plot or
Galbraith plot
 Based on heterogeneity test result, perform Random or
Fixed effect main meta analysis ; otherwise deal with
it using:
 subgroup analysis
 meta regression
 sensitivity analysis

Gedefaw D. 5/31/2023
 Statistical models for meta-analysis
38

1. Fixed-effect model

2. Random-effects model

Gedefaw D. 5/31/2023
 Fixed effect model
39

We assume that:
 There is one true effect size(single common (or “fixed’)
effect) that underlies all the studies in the analysis

 all differences in observed effects are due to sampling

error

 there is a single underlying population

 No statistical heterogeneity

Gedefaw D. 5/31/2023
 Fixed effect model
40

 It makes the assumption that there is a common odds ratio

across all studies
 Study weights are functions of within-study variance; and
confidence intervals are relatively narrow (liberal)
 Fixed effect model should be used when no much
evidence of heterogeneity

 In many practical settings, this assumption is not justified,

and must be replaced with a random effects model

Gedefaw D. 5/31/2023
 Random effects models
41

 DerSimonian(1986) developed a method for dealing with meta-

analysis of random effects.

 It incorporates an assumption that the different studies are

estimating different, yet related, intervention effects.

 It is based on the inverse-variance approach, making an

adjustment to the study weights according to the extent of
variation, or heterogeneity, among the varying intervention effects

 Example : metan command

 Has the option to do both fixed and random effects meta-
analyses

Gedefaw D. 5/31/2023
 Random effects models
42

 The effect size varies from study to study.

 For example
 The effect size might be higher (or lower) in studies where the
participants are older, or more educated, or healthier than in
other studies, or when a more intensive variant of an
intervention is used

Gedefaw D. 5/31/2023
 Random effects models
43

 Results in wider confidence intervals (conservative)

 Should be employed when there is evidence of heterogeneity

Gedefaw D. 5/31/2023
 Difference between the two model
44

I. The confidence interval for the summary effect is

wider under the random-effects model.

II. The estimate of the effect size differs under the two
models.

The random-effects method and the fixed-effect method

will give identical results when there is no heterogeneity
among the studies.

Gedefaw D. 5/31/2023
 Random effects models
45

 2 sources of variation:

 within-study variability

 between-study variance
 The variation in the true effect sizes between the
included studies (ie, heterogeneity).

Gedefaw D. 5/31/2023
 Heterogeneity
 Refers to the variation in study outcomes between
studies

 To pool an estimate from aggregate data ,first we

have to check whether the studies are homogeneous
or not

 If studies are significantly heterogeneous, pooling

estimates from such data will give unreliableestimates
 Heterogeneity
If there is heterogeneity, we should deal with the
reasons of heterogeneity
 Heterogeneity may be due to:

 variation in the population of primary studies

 intervention used

 outcome measured

 difference in study design

 methodological quality, etc.

Heterogeneity
Difference between studies

Favours treatment Favours control

No effect
 Detecting Heterogeneity
49

Gedefaw D. 5/31/2023
 Detecting Heterogeneity
50

1. Q statistic
 Quantify the total variation

where
 Yi is the effect size in study i,

 wi is the weight of the study defined as the inverse of its

variance (1/vi),
 M is the summary effect size of the meta-analysis of k studies

Gedefaw D. 5/31/2023
 Detecting Heterogeneity
51

2. I-square statistics
 quantifying relative heterogeneity

Gedefaw D. 5/31/2023
Tests of Heterogeneity
 Measure extent to which observed study outcomes differ
from calculated study outcome

 Visually inspect Forest Plot. Size of CI

 2 Test for homogeneity or Q Test can be used

 Weight of each study
53

 The weights to each study based on the inverse of the

overall study error variance (that is, 1/variance)

 Studies with a precise estimate of the population effect

size (a low variance) are assigned more weight, while
studies with a less precise estimate of the population
effect size (a high variance) are assigned less weight.

Gedefaw D. 5/31/2023
 Forest plot
54

 Three lines represent the odds ratios and confidence intervals

for the individual studies.

 The box at the center of each line represents the point

estimate of the odds ratio, and its size represents the weight
given to the odds ratio in estimating the pooled odds ratio.

 The center of the diamond represents the pooled point

estimate of the odds ratio.

 The left and right vertices of the diamond represent the

confidence interval for the pooled odds ratio.

Gedefaw D. 5/31/2023
 Forest plot
55

 Two vertical lines represent the null hypothesis OR= 1

(solid line) and the pooled point estimate of the odds
ratio (dashed line).

 If the diamond does not overlap the solid line, we have

rejected the null hypothesis of no treatment effect.

 A confidence interval that does not overlap the dashed

line indicates the corresponding study is inconsistent with
the pooled estimate of the odds ratio
Gedefaw D. 5/31/2023
 Publication bias
56

 A tendency for studies to be published based on their

results.
 A meta-analysis that relies only on published results
runs the risk of incorporating this bias into its own
findings.

Gedefaw D. 5/31/2023
 Publication Bias

 Studies with significant results are more likely

 to be published
 to be published in English
 to be cited by others
 to produce multiple publications
 Including only published studies can introduce publication
bias
 Methods for detecting publication bias:
 Graphical: funnel plot asymmetry
 Tests: Egger test, Rosenthal’s Fail-safe N
 Detecting Publication bias
 Funnel plot
Publication bias exists (asymmetrical)
Publication bias doesn’t exists (symmetrical)
- For continuous data- Effect size plotted vs. SE or sample size
- For dichotomous data- LogOR or RR vs. logSE or sample size

- Fail Safe Number (F)

Z= (∑ ES/1.645)2-N: (where N= no of papers)
 Publication bias
59

 Egger et al. recommend a graphical technique called a funnel

plot.
 A funnel plot is a scatterplot of studies in which the estimated
effect size is plotted on the horizontal axis and the reciprocal
of the standard error (precision) of the estimate is plotted on
the vertical axis.
 In the case of odds ratios, ln(OR) is plotted on the horizontal
axis, and SE(ln(OR)) is plotted on the vertical axis.
 If there is no publication bias, the plot should be symmetric
left-to-right, looking like an inverted funnel.
 If there is publication bias, the plot can be expected to be
asymmetric

Gedefaw D. 5/31/2023
 Publication bias
60

 There are two formal tests for symmetry of the funnel

plot

 There is also a technique called trim and fill for

adding studies to a funnel plot until it becomes
symmetric.

 In the trim phase, studies are removed if they lie

beyond 95% confidence limits for what the funnel is
estimated to be.

Gedefaw D. 5/31/2023
 Subgroup analysis
 Subgroup analysis
 Some participants, intervention or outcome you thought
were likely to be quite different to the others

 Should be specified in advance in the protocol

 Only if there are good clinical reasons

 Example subgroup analysis by design

Taken from Egger, M. et al. BMJ 1998;316:140-144

 Sensitivity Analysis
 Exclude and/or include individual studies in the
analysis

 Establish whether the assumptions or decisions we

have made have a major effect on the results of the
review

 ‘Are the findings robust to the method used to

obtain them?’
 Detecting Publication bias
 Funnel plot
Publication bias exists (asymmetrical)
Publication bias doesn’t exists (symmetrical)
- For continuous data- Effect size plotted vs. SE or sample size
- For dichotomous data- LogOR or RR vs. logSE or sample size

- Fail Safe Number (F)

Z= (∑ ES/1.645)2-N: (where N= no of papers)
 Funnel plot with pseudo 95% confidence limits

0
.2
se(SMD)

.4
.6
.8

-2 -1 0 1 2
Standardized mean difference (SMD)

Funnel plot with pseudo 95% confidence limits

0
.2
.4
se(SMD)

.6
.8
1

-2 -1 0 1 2
Standardized mean difference (SMD)
Correcting for publication bias
 Trim and fill method : tail of the side of the funnel plot with
smaller trials chopped off)
 Fail safe N
 Modelling for the probability of studies not published
 Influence analysis

 NB: there is no definite answer for assessing the presence

of publication bias
Conclusions

 Different methods lead to different conclusions

 Meta-analysis need effort but more valid

 Quality ratings can be tricky and controversial

 Pooled analysis
69

 A pooled analysis goes one step further than a

meta-analysis.
 Instead of combining the summary results (OR or RR)
from a number of different studies, the investigator
obtains copies of the raw data from the original
studies and re analyses them.

Gedefaw D. 5/31/2023
 Risk Vs Odds
70

Are they similar

Gedefaw D. 5/31/2023
 How to convert
71

Gedefaw D. 5/31/2023
 How to transform effect estimate
72

Computing relative risk from an odds ratio

 Risk ratios are easier to interpret than odds ratio

We may use formula or calculator

Example https://clincalc.com/Stats/ConvertOR.aspx

Gedefaw D. 5/31/2023
 How to transform effect estimate
73

We may use formula

ACR= Assumed Comparator Risk or baseline risk

Gedefaw D. 5/31/2023
Heterogeneity

 Is it appropriate to combine or pool results from various

studies?

 Different methodologies?

 Different outcomes measured?

 Problem greater in observational than clinical studies

Heterogeneity
Difference between studies

Favours treatment Favours control

No effect
Tests of Heterogeneity
 Measure extent to which observed study outcomes differ
from calculated study outcome

 Visually inspect Forest Plot. Size of CI

 2 Test for homogeneity or Q Test can be used
 low power (use p < 0.1 or 0.2)
Insufficient Power
Studies too small to detect any effect

Favours treatment Favours control

No effect
Meta-analysis
 Overall summary measure is a weighted average of
study outcomes.
 Weight indicates influence of study
 Study on more subjects is more influential
 CI is measure of precision
 CI should be smaller in summary measure
79

 Each study is assigned a weight based on the

amount of information it provides (e.g. the inverse
of the standard error of the OR) and
 In general larger studies have greater weight

Gedefaw D. 5/31/2023
80

 A weighted average of the individual study results

can then be calculated.
 The assumption underlying this analysis is that
 all of the studies are estimating the same
underlying effect and
 any variation between their results is due to chance.

 If their results are very different (i.e. they are

heterogeneous) then this assumption may not be true
and it might not be appropriate to combine the
results.
Gedefaw D. 5/31/2023
 Subgroup analysis

• Subgroup analysis
• Some participants, intervention or outcome you thought were
likely to be quite different to the others
• Should be specified in advance in the protocol
• Only if there are good clinical reasons
• Two types
• Between trial – trials classified into subgroups
• Within trial – each trial contributes to all subgroups
 Example subgroup analysis

Taken from Egger, M. et al. BMJ 1998;316:140-144

Sensitivity Analysis
 Exclude and/or include individual studies in the analysis
 Establish whether the assumptions or decisions we have
made have a major effect on the results of the review
 ‘Are the findings robust to the method used to obtain
them?’
Meta-analysis
 There are 2 statistical models/methods used in a
meta-analysis
 Fixedeffects model
 Random effects model
Fixed Effects Model
 “The model assumes that the true effect of treatment is
the same for every study.”
 All included studies measure same outcome
 Assume any difference observed is due to chance
 no inherent variation in source population
 variation within study, not between studies

 Inappropriate where there is heterogeneity present

 CI of summary measure reflects variability between patients
within sample
Random Effects Model
 “The model assumes that the true effect estimate for
each study vary.”
 Assumed studies are different and outcome will fluctuate
around own true value
 true values drawn randomly from population
 variability between patients within study and from differences
between studies
 Overall summary outcome is estimate of mean from which
sample of outcomes was drawn
 More commonly used with observational studies due to
heterogeneity
Random Effects Model
 Summary value will often have wider CI than with fixed
effects model
 Where no heterogeneity results of two methods will be similar
 If heterogeneity present may be best to do solely narrative
systematic review
 Strengths of Meta-Analysis
1. An excellent way to summarize research findings

2. More sophisticated than “vote-counting” procedures

3. Capable of finding effects that are obscured in other types of

reviews

4. Provides an organized way to summarize findings across a

large number of studies
 3. Publication Bias
When
 Studies with significant results are more likely

 to be published
 to be published in English
 to be cited by others
 to produce multiple publications
 Including only published studies can introduce publication bias

 Methods for detecting publication bias:

 Graphical: funnel plot asymmetry

 Tests: Egger test, Rosenthal’s Fail-safe N
Why does PB matter?

 Distorts the scientific record

 Hides the “truth”

 Influences doctors’ decision making

 Misleads policy makers

 Causes harm to patients

 Costly for the health service

 A form of scientific and research misconduct

 Sources of Bias
 Bias arising from the studies included in the review

 Bias arising from the way the review is done

 Publication bias is only one of the possible reasons for

asymmetrical funnel plot
 Detecting Publication bias
 Funnel plot
Publication bias exists (asymmetrical)
Publication bias doesn’t exists (symmetrical)
- For continuous data- Effect size plotted vs. SE or sample size
- For dichotomous data- LogOR or RR vs. logSE or sample size

- Fail Safe Number (F)

Z= (∑ ES/1.645)2-N: (where N= no of papers)
 Funnel plot with pseudo 95% confidence limits

0
.2
se(SMD)

.4
.6
.8

-2 -1 0 1 2
Standardized mean difference (SMD)

Funnel plot with pseudo 95% confidence limits

0
.2
.4
se(SMD)

.6
.8
1

-2 -1 0 1 2
Standardized mean difference (SMD)
Correcting for publication bias
 Trim and fill method : tail of the side of the funnel plot with
smaller trials chopped off)
 Fail safe N
 Modelling for the probability of studies not published
 Influence analysis

 NB: there is no definite answer for assessing the presence

of publication bias
Conclusions

 Different methods lead to different conclusions

 Meta-analysis need effort but more valid

 Quality ratings can be tricky and controversial

 Pooled analysis
97

 A pooled analysis goes one step further than a

meta-analysis.
 Instead of combining the summary results (OR or RR)
from a number of different studies, the investigator
obtains copies of the raw data from the original
studies and re analyses them.

Gedefaw D. 5/31/2023
98

Gedefaw D. 5/31/2023