· If you are viewing this slideshow within a browser window, select

File/Save as… from the toolbar and save the slideshow to your
computer, then open it directly in PowerPoint.
· When you open the file, use the full-screen view to see the
information on each slide build sequentially.
· For full-screen view, click on this icon at the lower left of your screen.

· To go forwards, left-click or hit the space bar, PdDn or  key.

· To go backwards, hit the PgUp or  key.
· To exit from full-screen view, hit the Esc (escape) key.
An Introduction to Meta-analysis
Will G Hopkins
Victoria University, Melbourne, Australia
· What is a Meta-Analysis?
· Definition, weighted average, heterogeneity, mixed-model meta-
regression
· Limitations to Meta-Analysis
· Individual differences or responses, publication bias
· How to Do a Meta-Analysis
· Generic measures, finding effects, study characteristics, study
quality, weighting factor, model, publication bias
· Summary and References
What is a Meta-Analysis?
· A systematic review of literature to address this question:
on the basis of the research to date, how big is a given effect,
such as…
· the effect of endurance training on resting blood pressure;
· the effect of bracing on ankle injury;
· the effect of creatine supplementation on sprint performance;
· the relationship between obesity and habitual physical activity.
· It is similar to a simple cross-sectional study, in which the
subjects are individual studies rather than individual people.
· But the stats are a bit harder.
· A review of literature is a meta-analytic review only if it
includes quantitative estimation of the magnitude of the effect
and its uncertainty (confidence limits).
· The main outcome is the overall magnitude of the effect.
· The effect in each study (the study estimates) and the meta-
analyzed mean outcome are often shown in a forest plot:
Study1
Study2
Study3
Study4
Study5
Data are means and
Study6 95% confidence
Study7 intervals.
Study8
Study9 Authors should show
Study10 magnitudes on the
Study11 plot.
Study12
Study13
Study14
Study15
Mean
harmful trivial beneficial
-2 -1 0 1 2 3
Effect on power output (%)
· The main outcome is not a simple average of the study
estimates.
· Meta-analysis uses the standard error of each study estimate to
give more weight to studies with more precise estimates.
· The standard error is an SD representing the expected variation
in the study estimate if the study was repeated again and again.
· The weighting factor is 1/(standard error)2.
· Other things being equal, use of this factor is equivalent to
weighting the effect in each study by the study's sample size.
· So, for example, a meta-analysis of 3 studies of 10, 20 and 30
subjects each amounts to a single study of 60 subjects.
· For controlled trials, this factor also takes into account
differences in standard error of measurement between studies.
· You can and you must allow for real differences or heterogeneity
in the magnitude of the effect between studies.
· In early (fixed-effects only) meta-analysis, you did so by testing for
heterogeneity using the Q or chi-squared statistic.
· The test has low power, so you used p<0.10 rather than p<0.05.
· If p<0.10, you excluded "outlier" studies and re-tested, until p>0.10.
· When p>0.10, you declared the effect homogeneous.
• That is, you assumed the differences in the effect between studies
were due only to sampling variation.
• Which made it easy to analyze the weighted mean effect.
· But the approach was unrealistic, limited, and suffered from the
problem of whether statistical non-significance means negligible.
· In random-effect meta-analysis, you accept there are always real
differences between all studies in the magnitude of the effect.
· The “random effect” is the standard deviation representing the
variation in the true magnitude from study to study.
 · You get an estimate of this SD and its precision.
• It is sometimes known as tau and is provided as tau2.
· The mean effect ± this SD is what folks can expect typically in
another study or if they try to make use of the effect.
· Instead of this SD, some researchers provide the Q and the related
I2 statistic, representing percent of variation due to real differences.
• Ignore any conclusions based on these uninterpretable statistics.
· Mixed-model meta-analysis or meta-regression is best of all.
· You include study characteristics as fixed effects.
· The study characteristics will partly account for differences in the
magnitude of the effect between studies.
• Example: differences between studies of athletes and non-athletes.
· The random effect now represents residual variation in the effect
between studies (i.e., not explained by the study characteristics).
· Include an extra random effect to account for repeated
measurement (multiple estimates) within studies.
· Custom software or an advanced package (e.g., SAS) is required.
Limitations to Meta-Analysis
· It's focused on mean effects and differences between studies.
· But what really matters is effects on individuals.
· So we should also quantify individual differences or responses.
· These can be expressed as standard deviations, but researchers
usually don't provide enough info to allow their meta-analysis.
· Inclusion of mean subject characteristics (e.g., age, gender,
genotype) as predictors in the meta-analytic model only partly
addresses this problem.
• It would be better if researchers made available all data for all
subjects, to allow individual patient-data meta-analysis.
· A meta-analysis reflects only published effects.
· But statistically significant effects are more likely to get published.
· Hence published effects are biased high.
· Funnel or related plots can be used to assess and reduce
publication bias.
How to Do a Meta-Analysis: Opt for a Generic Measure
· You can combine effects from different studies only when they
are expressed in the same units.
· In most meta-analyses, the effects are converted to a generic
dimensionless measure.
Main measures:
· Standardized difference or change in the mean (Cohen's d)
• Other forms are similar or less useful (Hedges' g, Glass's )
• But it’s better to standardize after the meta-analysis.
· Percent of range (max minus min) for psychometrics.
· Percent or factor difference or change in the mean
· Correlation coefficient and slope (seldom meta-analyzed)
· Risk, proportion, odds, hazard and count ratios.
Standardized Difference or Change in the Mean
· Express the difference or change in the mean as a fraction of
the between-subject standard deviation (mean/SD).
· Also known as Cohen's d (d stands for difference).
· This example of the effect of a treatment on strength shows
why the SD is important:
Trivial effect (0.1x SD) Very large effect (3x SD)
post post
pre pre

strength strength

· mean/SD is biased high for small sample sizes and needs

correcting before including in the meta-analysis.
· A problem with standardization:
· Study samples are often drawn from populations with different
SDs, so some differences in effect size between studies will be
due to the differences in SDs.
· Such differences are irrelevant and tend to mask more
interesting differences.
· The solution:
· Meta-analyze a better generic measure reflecting the biological
effect, usually percent or factor differences or changes.
• Rarely, the raw measure is best; for example, joint angles
representing flexibility.
· Combine the between-subject SDs from the studies selectively
and appropriately, to get one or more population SDs.
· Express the overall effect from the meta-analysis as a
standardized effect using this/these SDs.
· This approach also effectively eliminates the correction for
sample-size bias with standardized effects.
Percent or Factor Difference or Change in the Mean
· Many effects can be expressed as a percent or multiplicative
factor that tends to have the same value for every individual.
· Example: effect of a treatment on performance is +2%, or a factor
of 1.02, regardless of the raw value of the performance.
· For such effects, percent or factor difference or change can be
the most appropriate generic measure in a meta-analysis.
· If all the studies have small percent effects (<10%), use percent
effects directly in the meta-analysis.
· Otherwise express the effects (and their standard errors) as
factors and log-transform them before meta-analysis.
· Back-transform the outcomes into percents or factors.
· Where relevant, calculate standardized differences or changes in
the mean using the log transformed effects and logs of factor SD.
· Measures that are already in percent units (e.g., psychometrics,
fat as %BM) should be analyzed without log transformation.
· Measures of athletic performance need special care.
· The best generic measure is percent change.
· But a given percent change in an athlete's ability to output power
can result in different percent changes in performance in different
exercise modalities.
· Example: a 1% change in endurance power output produces the
following changes…
• 1% in running time-trial speed or time;
• ~0.4% in road-cycling time-trial time;
• 0.3% in rowing-ergometer time-trial time;
• ~15% in time to exhaustion in a constant-power test.
· So convert all published effects to changes in power output.
• A difficult and time-consuming task; you have been warned!
• See recent meta-analyses by my students and colleagues.
· For team-sport fitness tests, convert percent changes into
standardized effects using SD for chosen sports.
Correlation Coefficient and Slope
· These measures of association between two numeric variables
are seldom meta-analyzed.
Performance
· Studies with small between-subject SD
r = 0.80
have small correlations, so correlation
suffers from a similar SD problem as
standardized effects.
r = 0.20
· Solution: meta-analyze the slope.
· The slope is biased low (degraded) Maximum O2 uptake
only by random error in the predictor.
· Adjust for this bias by dividing the slope by the short-term reliability
intraclass correlation coefficient.
· Express the meta-analyzed slope as either…
• a correlation using SD for an appropriate population, or
• the effect of two SD of the predictor in that population.
Risk, Odds, Hazard and Count Ratios
· When the dependent variable is a proportion or count of
something, effects should be expressed as ratios.
· Risk ratio, relative risk, proportion ratio…
· Example: if proportions of inactive and active adults who get heart
disease after 20 years are 25% and 10%, risk ratio = 25/10 = 2.5.
· If proportions are time-independent classifications, convert all
effects to odds ratios for meta-analysis.
· Convert meta-analyzed odds ratios back into proportion ratios by
choosing a sensible proportion for the reference group.
• Use proportion ratio = OR/[1+p1(OR-1)], where OR is the odds ratio
and p1 is the reference proportion.
· Hazard ratio is the risk ratio for new occurrences in the next brief
instant of time (the "right-now" risk ratio).
· If proportions change with time, the proportion and odds ratios also
change, but the hazard ratio usually doesn't.
 · So, to meta-analyze studies with different time periods, convert any
proportion and odds ratios to hazard ratios.
• Assume p = (1-e-h.t), where p is the proportion and h is the hazard.
• Therefore hazard ratio h2/h1 = log(1-p2)/log(1-p1).
· But don't convert odds ratios from time-dependent case-control
studies, because (amazingly) these are already hazard ratios.
· If proportions in the two groups in all studies are low (<10%), all
proportion, odds and hazard ratios are effectively equal and
need not be interconverted.
· Express effects on counts as count ratios.
· Express standard errors of ratio effects as ×/÷ factor errors, then
log transform the ratios and errors for meta-analysis.
· Some meta-analysis programs do the log transformation and work
out the standard errors for you.
• Researchers make big mistakes with these programs!
How to Do a Meta-Analysis: Find and Record Effects
· Search the literature for studies of a specific effect.
· If the effect has been meta-analyzed already, you can do another,
if it was done badly or if there are many new studies.
· As you assemble the published papers, broaden or narrow the
focus of your review to make it manageable and relevant via…
• …design (e.g., only randomized controlled trials), population (e.g.,
only competitive athletes), or treatment (e.g., only acute effects).
• Minimum sample size is ~10 studies; many more needed for good
analysis of modifying effects of study characteristics.
· Document your searches, inclusions and exclusions.
· Record each effect magnitude and inferential information
(sample size, p value, confidence limits, SD of change scores).
· Convert effects into values on a single scale of magnitude.
· In studies with more than one group, record the effect in each group
and include relevant fixed and random effects in the analysis.
How to Do a Meta-Analysis: Get Study Characteristics
· Record study characteristics that might account for differences
in the effect magnitude between studies.
· Include the study characteristics as covariates in the meta-
analysis. Examples:
· duration or dose of treatment;
· method of measurement of dependent variable;
· quality score;
· gender and mean characteristics of subjects (age, status…).
• Record separate outcomes for females and males from the same
study, if possible.
• Otherwise analyze gender as a proportion of one gender; for
example, in a study of 3 males and 7 females, “Maleness” = 0.3.
• Use this approach for all problematic dichotomous characteristics
(sedentary vs active, non-athletes vs athletes, etc.).
· group identity (e.g.: control, experimental)
How to Do a Meta-Analysis: Assess Study Quality?
· Most meta-analysts score the quality of a study.
· Examples (scored yes=1, no=0):
• Published in a peer-reviewed journal?
• Experienced researchers?
• Research funded by impartial agency?
• Study performed by impartial researchers?
• Subjects selected randomly from a population?
• Subjects assigned randomly to treatments?
• High proportion of subjects entered and/or finished the study?
• Subjects blind to treatment?
• Data gatherers blind to treatment?
• Analysis performed blind?
· Use the score to exclude some studies, and/or…
· Include as a covariate in the meta-analysis, but…
· Some statisticians advise caution when using quality scores.
How to Do a Meta-Analysis: Get the Weighting Factor
· Calculate the standard error for each effect via one or more of…
· the confidence interval or limits
· the test statistic (t, 2, F)
· the p value
• If the exact p value is not given and you can't calculate the standard
error from the data, try contacting the authors for it.
• Otherwise, if "p<0.05", analyze as p=0.05.
• If "p>0.05" with no other info, deal with the study qualitatively.
· the SD of change scores (for controlled trials)
• For studies lacking sufficient information to calculate standard
errors, calculate the typical error in every other study, impute typical
errors, then estimate standard errors via SD of change scores.
• The spreadsheet for sample-size estimation at Sportscience
converts p values or confidence limits to typical and standard errors
· the data: done for you by the software, depending on the effect.
• But be careful to choose the right options.
How to Do a Meta-Analysis: Develop the Model
· Do a mixed-model meta-regression.
· Estimate and interpret Effect on power output (%)
the effect for interesting 3
types of study or mean
subject. 2
· Show a moderator plot

beneficial
rather than a forest plot: 1
· In a mediator plot, the
subject characteristic
trivial
0
is a difference score
or change score.
-1
harmful

Data are means and -2

90% confidence 0 5 10 15 20
intervals Baseline training (h.wk-1)
· For any linear covariate, estimate and interpret the effect of 2x
the average of between-subject SD from appropriate studies.
· Double the SD representing the between-study random effect to
interpret its magnitude as the unexplained typical differences in
the magnitude of the effect between settings.
· The random effect provided as tau2 may be in log-transformed
units and may need back-transformation to a factor SD.
• Apply the p value for tau2 directly to the SD to get confidence limits.
· For effects where there are control or other reference groups…
· include each group effect separately, if possible;
· include a within-study random effect to account for the resulting
repeated measurement;
· and of course, include a fixed effect to estimate the uncontrolled
effect and the effect relative to control.
· Inspect between-subject SD between and within studies for
evidence of individual differences or responses.
How to Do a Meta-Analysis: Deal with Publication Bias
· Some meta-analysts present the effect magnitude of all the
studies as a funnel plot, to address the issue of publication bias.
· Published effects tend to be larger than true effects, because...
• effects that are larger simply because of non-sig. SE published
sampling variation have smaller p values, missing
studies delete
studies
• and p<0.05 is more likely to be published. funnel of funnel
of all
· A plot of standard error vs effect magnitude allstudies
non-sig
if studies if
should have a triangular or funnel shape. effect=0 effect >0
· But if some non-significant studies weren’t effect 0 value with
published, the plot will be asymmetrical. magnitude huge sample
• The missing studies are generally smaller, therefore have larger SE.
· So delete studies with larger SE to give a symmetrical plot.
· However, effect heterogeneity also disrupts the funnel shape,
making it hard to decide where to draw the line to delete studies.
· A plot of standardized residuals (random-effect solution) vs
standard error works much better. But it requires skill with SAS.
Summary
· Meta-analysis is a statistical review of the magnitude of an effect.
· Meta-analysis uses the magnitude of the effect and its precision
from each study to produce a weighted mean.
· Traditional meta-analysis is based unrealistically on using a test
for heterogeneity to exclude outlier studies.
· Random-effect (mixed-model) meta-analysis estimates
heterogeneity and allows estimation of the effect of study and
subject characteristics on the effect.
· For the analysis, the effects have to be converted into the same
units, usually percent or other dimensionless generic measure,
but do not meta-analyze standardized effects.
· It's possible to account for publication bias and identify outlier
studies using a funnel plot or residuals plot.
References
· Recent meta-analyses co-authored by me. Also my article on improving
meta-analyses http://sportsci.org/2018/metaflaws.htm.
· A good source of meta-analytic wisdom is the Cochrane Collaboration, an
international non-profit academic group specializing in meta-analyses of
healthcare interventions.
· Website http://www.cochrane.org
· Publication: Cochrane Handbook for Systematic Reviews of
Interventions http://training.cochrane.org/handbook.
· But the (free) Cochrane meta-analysis software is somewhat limited.
These references are getting out of date:
· Simple reference: Bergman NG, Parker RA (2002). Meta-analysis: neither quick nor easy.
BMC Medical Research Methodology 2, http://www.biomedcentral.com/1471-2288/2/10.
· Glossary: Delgado-Rodríguez M (2001). Glossary on meta-analysis. Journal of Epidemiology
and Community Health 55, 534-536.
· Reference for problems with publication bias: Terrin N, Schmid CH, Lau J, Olkin I (2003).
Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine 22,
2113-2126.
This presentation is available from:

See Sportscience 8, 2004