Review doi:10.

1038/nature25753

Meta-analysis and the science of

research synthesis
Jessica Gurevitch1, Julia Koricheva2, Shinichi Nakagawa3,4 & Gavin Stewart5

Meta-analysis is the quantitative, scientific synthesis of research results. Since the term and modern approaches to
research synthesis were first introduced in the 1970s, meta-analysis has had a revolutionary effect in many scientific
fields, helping to establish evidence-based practice and to resolve seemingly contradictory research outcomes. At the
same time, its implementation has engendered criticism and controversy, in some cases general and others specific
to particular disciplines. Here we take the opportunity provided by the recent fortieth anniversary of meta-analysis
to reflect on the accomplishments, limitations, recent advances and directions for future developments in the field of
research synthesis.

S
ynthesizing results across studies to reach an overall understanding Evolution; https://osf.io/g65cb), and in better training for practitioners
of a problem and to identify sources of variation in outcomes is an and reviewers in the rationales and methodologies of meta-analyses and
essential part of the scientific process. Until recently, the results of systematic reviews.
scientific studies have been summarized in narrative reviews. However, Here we highlight some of the main principles and characteristics
this approach becomes inadequate when there are hundreds of studies on of high-quality meta-analytic methodology and briefly summarize the
a given research question1,2, and the difficulties of carrying out narrative development of the field. We also discuss the limitations, utility and
reviews to identify and summarize evidence in a transparent and objec- achievements of meta-analysis in several fields and, as a case study, its
tive manner have become increasingly apparent as research results have role in advances in ecology, evolutionary biology and conservation (EEC).
mushroomed across scientific fields3. Finally, we address several recent criticisms of the meta-analytic approach
During the past few decades, scientifically rigorous systematic reviews and suggest ways in which future developments in research synthesis
and meta-analyses, carried out following formal protocols to ensure could facilitate the most rapid progress in the fields in which it is used.
reproducibility and reduce bias, have become more prevalent in a range of
fields1 (Box 1). Systematic reviews aim to provide a robust overview of the Meta-analyses use well-documented methodologies
efficacy of an intervention, or of a problem or field of research. They can Systematic reviews aim to be transparent, reproducible and updatable,
be combined with quantitative meta-analyses to assess the magnitude of and to address well-defined questions. The systematic review process
the outcome across relevant primary studies and to analyse the causes of includes the use of formal methodological guidelines for the literature
variation among study outcomes (effect sizes). Narrative reviews remain search, study screening (including critical appraisal of eligible studies
useful for exploring the development of particular ideas (as we do here) according to pre-defined criteria), data extraction, coding and often
and for advancing conceptual frameworks, but they cannot accurately ­statistical analysis (that is, meta-analysis), along with detailed, transparent
summarize results across studies4. documentation of each step. Software, protocols and reporting guidelines
Four decades after its introduction, we are seeing widespread main- for systematic reviews and meta-analyses are well established in many
stream acceptance of meta-analysis as a research synthesis tool, but also fields; for example, PRISMA (Preferred Reporting Items for Systematic
the signs of what may be considered a ‘midlife crisis’ as it has begun the Reviews and Meta-Analyses12; http://www.prisma-statement.org/) is “an
transition to a mature field. While the number of published meta-anal- evidence-based minimum set of items for reporting in systematic reviews
yses has continued to increase rapidly, too many meta-analyses and and meta-analyses” and includes a checklist of 27 items and a template
­systematic reviews are of low quality5–7. The publication of methodo- flow chart for the presentation of a systematic review (a ‘PRISMA flow
logically flawed meta-analyses indicates that peer reviewers, editors diagram’; Fig. 1a). Guidelines for developing and preparing systematic
and authors are not fully aware of or are indifferent to the large body of review protocols are published in PRISMA-P (http://www.prisma-­
well-developed meta-analytic methodology, and that reviewers might feel statement.org/Extensions/Protocols.aspx)13.
unqualified to address statistical issues. Low-quality meta-analyses have If the systematic review reveals sufficient and appropriate quantitative
attracted strong criticism5,8 and even calls for a halt in publication of all data from the studies that are being summarized, then a meta-analysis can
meta-analyses9. Although it is certainly both valid and valuable to criticize be conducted. In a meta-analysis, one or more outcomes in the form of
poor methodo­logy and reporting, such criticism should result in a call for effect sizes are extracted from each study. Effect sizes are designed to put
improved standards (as for pre-clinical trials10) rather than abandonment the outcomes of the different studies being combined on the same scale,
of the field11. We believe that the solution lies in the rigorous application using a suite of metrics14,15 that includes odds and risk ratios, standardized
of stricter methodological and reporting quality criteria for publishing mean differences, z-transformed correlation coefficients and logarithmic
meta-analyses (see, for example, Tools for Transparency in Ecology and (‘log’) response ratios. It is essential for the effect-size metric used to be

1
Department of Ecology and Evolution, Stony Brook University, Stony Brook, New York 11794-5245, USA. 2School of Biological Sciences, Royal Holloway University of London, Egham, Surrey,
TW20 0EX, UK. 3Evolution and Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, New South Wales 2052, Australia.
4
Diabetes and Metabolism Division, Garvan Institute of Medical Research, 384 Victoria Street, Darlinghurst, Sydney, New South Wales 2010, Australia. 5School of Natural and Environmental
Sciences, Newcastle University, Newcastle upon Tyne, NE1 7RU, UK.

8 m a rc h 2 0 1 8 | VO L 5 5 5 | NAT U R E | 1 7 5
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.
 RESEARCH Review

Box 1
A brief history of meta-analysis
The first formal attempt to combine information from multiple
sources (see figure) was made in 1904 by K. Pearson83 with
the aim of ascertaining the effectiveness of vaccination in
preventing soldiers from contracting typhoid. R. A. Fisher, another
important figure in the development of modern statistical science,
subsequently introduced a method for combining probabilities
from different studies84. In the late 1930s, W. Cochran and
F. Yates described approaches that were essentially the same as
modern fixed-effect and random-effects models85, which were
later formalized and generalized by Cochran86. However, it was
not until the insight of psychologists G. Glass and M. Smith in
1977—that outcome measures from different experiments could
be standardized and put on the same scale87—that meta-analysis
began to affect scientific research to a large extent. Meta-analysis
was initiated almost simultaneously in medicine and the social
sciences88 and was initially met in all fields with a combination of
enthusiasm and condemnation52,88. Methodology was formalized
and developed in the two decades following 1977 in multiple
fields16,89–91, with influential studies spreading from medical and
social sciences to EEC in the early 1990s23,92 (Table 1).
Rapid methodological and procedural developments have
followed, with cross-disciplinary interactions being the key drivers
of progress. The introduction of electronic literature databases
and journal articles was central to the development of current
practices; a lack of access in poorer institutions and countries
hinders scientific progress. The highly interdisciplinary Society
for Research Synthesis Methodology (http://www.srsm.org/) was
established in 2005, after which it began publication of Research
Synthesis Methods. The large collaborative networks the Cochrane
Collaboration (established in 1993; now known as Cochrane;
https://www.cochrane.org) and the Campbell Collaboration Box 1 Figure | Milestones in the history of meta-analysis. The red
(established in 1999; https://www.campbellcollaboration.org) line shows the number of papers from a Scopus search; the dashed
oversee systematic reviews in the medical and social sciences, component indicates the expected future trajectory. The milestone
respectively, bringing practitioners and methodologists together publications12,16,17,19,74,83,86,87,95–100 are chosen on the basis of two
and setting standards for research-synthesis publications and main criteria—precedence and influence (for these criteria, we relied
evidence-based guidelines for practice and policy. heavily on refs 93 and 94).

readily interpretable, scientifically meaningful and comparable among and sources of heterogeneity (see sections ‘Meta-analysis is essential for
meta-analyses, and for its sampling distribution to be known, so that progress in science’ and ‘Meta-analysis in EEC as a case study’). Although
statistical models can be constructed appropriately. these goals differ considerably among disciplines, quantifying hetero­
The effect sizes are then entered into a statistical model with the goal geneity is universally important.
of assessing overall effects and heterogeneity in outcomes. These models Heterogeneity tests and meta-regressions both use weighting based
are based on an assumption of either a common effect (‘fixed effect’) or on the precision of the estimate of the effect: larger studies with higher
random effects (Fig. 1b)16. The common-effect (or fixed-effect) model precision are weighted more heavily than smaller and/or more variable
assumes that variation in effect sizes among studies is due to within-study studies18 (Fig. 1b, d). There are many issues to consider in ­constructing
(sampling) variance and that all studies share a common ‘true’ effect. The these statistical models, including appropriate weighting and how to
random-effects model assumes that, in addition to sampling variance, the account for non-independence (see sections ‘Meta-analysis in EEC as a
true effects from different studies also differ from one another, represent- case study’ and ‘Limitations, controversies and challenges’). In addition,
ing a random sample of a population of outcomes, and is analogous to a tools have been developed for evaluating publication bias and power and
random-effects model in an analysis of variance (ANOVA). Thus, ran- for conducting sensitivity analyses19–21 (Fig. 1e, f).
dom-effects models include an extra variance component to account for
between-study variance (heterogeneity) in addition to within-study vari- Meta-analysis is essential for progress in science
ance. Common-effect models are based on the assumption that the results Meta-analysis has generally been used with two different fundamental
apply only to a given group of studies. Random-effects models apply more goals in mind, resulting in the use of contrasting approaches. The first
generally. In carrying out a meta-analysis, the central tendency (the mean) of these goals is to assess the evidence for the effectiveness of specific
and its confidence limits are evaluated, as well as the heterogeneity in the interventions for a particular problem or hypothesized causal associ-
effect across studies. To identify the magnitude and sources of variation in ations for a condition, often over a relatively small number of studies
effect size among studies (Fig. 1c), earlier studies relied on simple hetero­ (fewer than about 25). The second, quite different, fundamental goal is
geneity tests16, whereas more recent work often uses meta-regressions17. to reach broad generalizations across larger numbers of study outcomes
The ‘main effect’ or ‘grand mean’ can be of critical importance or largely (dozens to ­hundreds) to provide a more comprehensive picture than can
irrelevant, depending on the goals of the meta-analysis and the magnitude be attained from an individual primary study. The differences in approach

1 7 6 | NAT U R E | VO L 5 5 5 | 8 m a rc h 2 0 1 8
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.
 Review RESEARCH

a b Study 1 e
Records from Records from
databases other sources Study 2
Identification

Study 3

Precision
Records after Study 4
duplicates removed
Study 5

Common-effect model
Records Records
Random-effects model
screened excluded
Screening

0 0
Effect size Effect size

Full-text papers Full-text papers f Study year 1

assessed excluded
+ Study year 2
Eligibility

+ Study year 3
c d
+ Study year 4
Studies included in + Study year 5
qualitative synthesis + Study year 6
Effect size

Effect size
(systematic review)
+ Study year 7
Included

+ Study year 8
0 0 + Study year 9
Studies included in + Study year 10
quantitative synthesis
(meta-analysis) X Y Z 0
Categorical moderator Continuous moderator Effect size
Figure 1 | Various charts and plots common to meta-analysis. are common in EEC and some social sciences. d, A ‘bubble’ plot showing
a, A PRISMA flow diagram12, which describes information flow (the a line predicted from a meta-regression analysis; the sizes of the bubbles
number of relevant publications) at the four stages of the systematic reflect the sample sizes of the individual studies. This type of plot may be
review process (‘identification’, ‘screening’, ‘eligibility’ and ‘included’). used to assess continuous predictors (such as publication year or length of
b, A ‘forest’ plot of the various means (symbol centres), confidence limits a treatment). e, A ‘funnel’ plot displays the effect size against the precision
(95% confidence intervals; whiskers) and precision (indicated by the with which it is estimated, which relates to its weight. Here we illustrate
size or ‘weight’ of the symbols, with larger symbols indicating greater data (red points, with the dotted red line indicating an overall effect) that
precision) of the effect-size determined from individual studies (black), display ‘funnel asymmetry’, which could indicate publication bias, along
and the overall means (symbol centres) and 95% confidence intervals with data (open circles) obtained after applying the trim-and-fill method,
(symbol widths) determined using meta-analysis with a common-effect a sensitivity analysis that corrects for a potential publication bias.
(or fixed-effect) model (brown) and a random-effects model (purple). This f, A ‘forest’ plot of a cumulative meta-analysis in which outcomes are
type of plot is used to represent effect sizes and their confidence intervals added into the analysis in chronological order, demonstrating an increase
graphically. c, A summary ‘forest’ plot of the mean effect sizes and 95% in precision and a convergence of effect sizes as studies are added, and a
confidence intervals for different groups of studies. This type of plot may temporal trend across studies. The dashed black lines in b–f indicate
be used to assess categorical moderators (denoted X, Y and Z here) and ‘no effect’ of an intervention on the outcome.

and goals affect not only the scale of meta-analyses, but every step of the In the second case, when the goal is to reach broad generalizations,
research synthesis, from study inclusion criteria to the statistical models the population of studies may be large and heterogeneous and, although
used. In both approaches, meta-analysis is used to synthesize evidence estimating the main effect of a particular phenomenon or experimen-
across studies to detect effects, to estimate their magnitudes and variation tal treatment may be important, identifying sources of heterogeneity in
and to analyse the factors (covariates or moderators) that influence them. outcomes is often central to understanding the overall phenomenon23.
When the goal is to assess evidence for specific interventions, the focus Meta-analyses undertaken with the aim of reaching broad generalizations
of meta-analyses is primarily on accurately estimating an overall mean deliberately incorporate results from heterogeneous populations so that
effect, and may include identifying factors that modify that effect. This broad generalizations and the factors that modify them can be exam-
approach is exemplified by the PICO (population, intervention, compar- ined and tested. This approach is common in the fields of EEC and in
ator, outcome) framework (and its extensions) for formulating questions, some social sciences, in which meta-analyses have been used to address
in which specification of these elements is central to the purpose of the fundamental problems, to weigh the evidence for prominent theories or
synthesis22, as it is, for example, when assessing clinical effectiveness or hypotheses and to consider the generality of common findings, observa-
the effectiveness of interventions in other disciplines. Question formu- tions or phenomena23,24.
lation using PICO has been adopted in a wide range of fields, including Of course, to some extent there is a continuum rather than an absolute
medicine and the social sciences. Although moderating factors might dichotomy in meta-analytic approaches, with overlap between disciplines.
be important for understanding how the overall effect is influenced by A limitation of using broad inclusion criteria is the difficulty in ade-
study or population characteristics, meta-analyses for which the primary quately accounting for high heterogeneity. A limitation of a reductionist
goal is to estimate the effects of a specific intervention accurately tend to scope and narrow focus is the limited inference that is possible outside
emphasize the consequences of that intervention for a specific p ­ opulation. of a n
­ arrowly specified population or for factors that modify outcomes,
This type of meta-analysis must clearly and specifically ­delineate the whereas the inclusion of a broader definition of the population of interest
­population in question. Consequently, the results may apply only to and potential factors that could affect outcomes might be highly ­revealing.
that population; for example, the conclusions of a research synthesis of Both approaches can be limited or even biased. A collection of many
a ­medical intervention based on studies that included only middle-aged narrowly focused reviews of what is essentially the same intervention
males might not apply to females or to younger males. can generate spurious results, as can the opposite approach of ‘fishing’ for

8 m a rc h 2 0 1 8 | VO L 5 5 5 | NAT U R E | 1 7 7
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.
 RESEARCH Review

significance among many hypothesized explanatory factors or covariates Table 1 | Development of systematic reviews and meta-analyses
in an excessively broad study. in EEC
For both of these basic goals (evaluation of specific interventions or Year Milestone
reaching a broad understanding of a general problem), meta-analysis 1991 First meta-analysis in ecology published78
provides a more powerful and less biased means for clarifying, quanti- 1995 Seminal paper by Arnqvist and Wooster79 published in
fying and disproving (or confirming) assumed wisdom than do conven- Trends in Ecology and Evolution, introducing meta-analysis to
tional approaches25 including narrative reviews and flawed quantitative many ecologists
methods such as ‘vote counts’ (see section ‘Limitations, controversies 1995 National Center for Ecological Analysis and Synthesis
­established in USA
and challenges’). Meta-analytic methods have resolved apparently incon- 1997 MetaWin, the first software for ecological meta-analysis
clusive data to arrive at a clearer picture, often more rapidly than other created46
approaches. In medicine, meta-analyses can unambiguously assess the 1999 Special feature on meta-analysis published in Ecology,
effectiveness of particular surgical or pharmaceutical interventions or the ­including an influential paper on statistical issues in
­ecological meta-analysis50 and the introduction of the
statistical significance of hypothesized causal associations. For example,
­logarithmic response ratio as a metric for effect size80
a meta-analysis of 12 clinical studies was able to demonstrate conclu- 2001 First general review of meta-analysis in ecology published81
sively a clear relationship between maternal obesity and risk of neural 2003 Centre for Evidence-Based Conservation established in UK
tube defects despite considerable variation in the effect sizes reported in 2007 Collaboration for Environmental Evidence created
individual studies (from a slightly greater incidence of these birth defects 2008/2009 Seminal papers on phylogenetic meta-analysis
for overweight mothers compared to normal-weight mothers, to three ­published43,45 and phyloMeta software for integrating
­phylogeny into meta-analyses released82
times the risk (odds ratio of 3.11) for severely obese mothers compared 2011 Environmental Evidence (the official journal of the
to normal-weight mothers)26. Similarly, primary studies of the value of a ­Collaboration for Environmental Evidence) established
family-based intervention approach for serious juvenile offenders called 2013 First handbook of meta-analysis in ecology and evolution
multi-systemic therapy were seemingly inconsistent; however, despite the published73
logical and theoretical basis for multi-systemic therapy, a meta-­analysis 2014 OpenMEE, software for ecological and evolutionary
­meta-analysis, released47
found no significant differences between it and conventional social 2016 First international conference of the Collaboration for
­services in the success of outcomes27. Both of these meta-analyses have ­Environmental Evidence, in Stockholm
had ramifications for evidence-based practice.
The most consequential effect of introducing formal research-
synthesis methodology has been a profound change in the way scientists services36,37, which demonstrated that declines in species richness have
think about the outcomes of scientific research. An individual primary negative effects on the functioning of ecosystems. It has been found38
study may now be seen as a contribution towards the accumulation that ecological restoration can reverse environmental degradation and
of evidence rather than revealing the conclusive answer to a scientific increase biodiversity and the provisioning of ecosystem services in a wide
problem25,28. There are certainly cases where a single revelatory study range of ecosystems globally, although not to full recovery compared to
has completely illuminated and resolved a major problem; however, in reference ecosystems.
many cases syntheses can provide a more general and complete picture Similarly, meta-analytic techniques have provided evolutionary biolo-
of the evidence than can any individual study. The results of initial gists the tools to test key hypotheses based on theories of natural selection,
studies are too often not confirmed by those of subsequent studies or by sexual selection and animal social behaviour at unprecedented scales35.
syntheses of a body of research. Additional major contributions of the Examples of prominent evolutionary meta-analyses include assessments
introduction of meta-analysis have been increased attention to reporting of correlations between measures of genetic diversity, fitness and popu-
standards in primary studies, including full and transparent reporting of lation size39. One conclusion is that a reduction in population size due
data and the recognition that studies that report no significant effect to habitat fragmentation reduces genetic variation, which in turn has a
are as potentially interesting and valuable as those that report low negative impact on fitness in the affected populations.
P values29,30. In EEC, meta-analytic techniques have greatly expanded the a­ bility
to construct large-scale overviews of study outcomes—over larger
Meta-analysis in EEC as a case study ­spatial scales, different time periods, multiple systems and a diversity of
Meta-analysis was first adopted by ecologists and evolutionary biolo- ­organisms that are beyond the scope of any one researcher or research
gists some 25 years ago (Table 1) and has had a considerable impact on group. For example, a global meta-analysis40 of almost 600 latitudinal
this research field in both fundamental and applied areas. Meta-analytic gradients in species diversity verified the high degree of generality of the
approaches in ecology were introduced at around the same time as it decline in diversity with latitude, but also identified important f­ actors
became increasingly urgent to provide accurate quantitative assessments, that modify this pattern. Meta-analysis has also been a valuable tool
predictions and practical solutions to pressing environmental issues for practitioners in EEC involved in collaborative research who wish to
such as biodiversity losses, the increase in invasive species and biotic ­combine original results from experiments carried out across multiple
responses to climate change. Meta-analysis has provided tools for sum- study sites41,42.
marizing evidence for these effects, their impacts and the effectiveness of Unlike clinical medicine and the social sciences, fields in which
­interventions. The increased use of meta-analyses and systematic reviews research focuses on a single species, the multi-species nature of much
in conservation and applied ecology has been facilitated by the promotion of EEC research and therefore of meta-analyses has led practitioners to
of evidence-based approaches in this field31,32, especially through organi- integrate phylogenetic comparative methods with meta-analytic models
zations such as the Centre for Evidence-Based Conservation (http://www. to take into account potential non-independence among lineages due to
cebc.bangor.ac.uk) and the Collaboration for Environmental Evidence shared evolutionary history43–45. Non-independence among outcomes
(https://www.environmentalevidence.org; Table 1). due to the variation among sources may be more obvious in EEC than
Applications of meta-analyses and, more recently, systematic reviews in other fields because of the large size and complex data structure of
in EEC have highlighted major gaps in research33, provided assessments many meta-analyses in EEC. However, non-independence is a ubiqui-
of the effects of major environmental drivers (such as climate change34) tous ­problem for research synthesis in most research fields, and much
and of the effectiveness of conservation and management strategies31, work remains to be done to better model and account for sources of
and enabled evaluations of the evidence for ecological and evolutionary non-independence.
theories35. Examples of influential ecological meta-analyses include quan- The structural characteristics of data in EEC and the goals of ­generality
tifications of the effects of biodiversity on ecosystem functioning and typically result in high heterogeneity. Rather than seeking to explain

1 7 8 | NAT U R E | VO L 5 5 5 | 8 m a rc h 2 0 1 8
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.
 Review RESEARCH

all of the heterogeneity among studies, the goal is often to identify key of these methods and in the application of arbitrary and less-well-­
factors of commonality—to detect the signals amid the noise when justified methodologies that are sometimes inaccurately referred to as
­gaining information about these hypothesized key factors is more meta-­analyses. The use of statistically flawed approaches can lead to
­important than achieving a clean accounting of all sources of variabil- erroneous and misleading results that masquerade as serious research
ity. This is a different perspective from that of meta-analyses that focus syntheses. The term meta-analysis should be applied only to studies that
­narrowly on, for example, detecting the efficacy of a specific intervention. use well-­established statistical procedures, such as appropriate effect-size
Advances in meta-analysis in EEC have been stimulated by many calculation, weighting and heterogeneity analysis57, and statistical models
­factors, including learning from practitioners in other disciplines, effec- that take into account the distinct hierarchical structure of meta-­analytic
tive and widespread short courses for students and practising ­scientists, data, or to studies that develop rigorously justified methodological
and the development of software that is tailored specifically to this advances of these methods. Unfortunately, the term is often misapplied
field46,47. Methodological innovations in meta-analytic techniques that to any study that uses data from several primary publications, regardless
have been incorporated or developed in EEC, in addition to phyloge- of the rigour of the ­methodology. Statistically flawed procedures such
netic approaches, include the meta-analysis of factorial experiments48, the as vote-counting, which provide only limited information about study
introduction and wide acceptance of randomization (permutation) tests outcomes, can be very misleading and have long been discredited, are still
in meta-analysis49, the early embrace of random-effects and mixed-effects used in published papers6,50. Vote-counting is a deceptively plausible and
models when they were still highly controversial in other disciplines50, appealingly convenient procedure whereby the generality of findings in
and methods for the inclusion of qualitative information such as expert a group of studies is assessed by counting up the number of significant
opinions51. and non-­significant results in individual studies (or by elaborations on
The introduction and incorporation of meta-analysis in ecological this approach). Although it is vulnerable to erroneous inferences and
research have raised similar objections to those raised in other disci- provides u ­ nreliable information on the magnitudes or heterogeneity of
plines, and these criticisms and others have been similarly refuted across effects, it persists, zombie-like, returning by the efforts of the naive or
­disciplines11. For instance, critics have claimed that the potential for publi- determinedly ignorant to haunt the scientific literature. Vote-counting is
cation bias in the literature (that is, the under-reporting of non-significant not a meta-analytic technique, and is not an acceptable basis for mean-
results or disconfirming evidence21) invalidates the use of meta-analysis. ingfully summarizing research results in published papers.
This objection has been refuted by research synthesists in many fields, Meta-analyses that are not weighted by inverse variances are common
who point out that when publication bias exists, it presents problems that and often poorly justified, and present different problems. Unlike vote-
are not unique to meta-analyses, but affect any attempt to summarize the counts, unweighted meta-analyses can be unbiased and may provide
results of the literature or to reach valid conclusions from it. In another information on the magnitude of the effects8. However, in an unweighted
instance, as in the early criticisms of meta-analysis in social sciences52, analysis, within- and between-study variation cannot be readily separated,
some ecologists have claimed that ecological studies are too heterogeneous and so common- and random-effects models cannot be used and hetero­
to be combined statistically in a meaningful way9 and that ecology is best geneity may be difficult to assess properly. Unweighted meta-analysis
served by accumulating a catalogue of case studies53. Analogously, the also increases the influence of small studies29, which have often been
basis for the early objections to introducing statistics to ecology in the found to report larger and more variable effects than those reported
mid-twentieth century was the inability to fully account for the unique- for larger s­ tudies (as a result of the smaller studies being more likely to
ness of individual organisms and the micro-site environmental variation suffer from random noise, and possibly publication bias). An alterna-
using means and statistical tests. Despite the criticism, the introduction of tive when variances are unavailable from primary studies is weighting
meta-analysis in EEC has been embraced enthusiastically by the m ­ ajority by sample size or other metric, but this method does not incorporate the
of scientists in these disciplines as a ‘remote sensing tool’ that helps information that an inverse-variance-weighted analysis provides and
scientists to generalize the findings of individual studies to reach a can introduce unknown biases. These problems are particularly acute
broader understanding11, and the number of meta-analyses published in with small s­ ample sizes. One argument that is often made in support of
EEC has increased exponentially over time54. unweighted meta-analysis is that the variances needed for a weighted
meta-analysis are frequently unavailable owing to poor data reporting in
Limitations, controversies and challenges the primary studies, and it is undesirable to leave studies with missing data
Despite its current utility and future potential, meta-analysis has v­ arious out of the meta-analysis. One possible solution is to use one of the various
limitations as a tool for research synthesis and for informing decisions. methods that have been developed for imputing or otherwise modelling
Meta-analyses and systematic reviews can highlight areas in which missing data. And, although data reporting practices are being improved
evidence is deficient, but they cannot overcome these deficiencies—they slowly, it may be that many older studies are simply inadequate for accurate
are statistical and scientific techniques, not magical ones. For example, quantitative reviews. Another argument for unweighted meta-
in a systematic review of the literature on hypotheses for explaining bio- analysis is that the meta-analysis simplifies to an essentially unweighted
logical invasions, a major gap was found33 in published studies on inva- analysis when between-study variation is much larger than within-study
sive species in the tropics, highlighting not only what is known but also variation58. However, a weighted meta-analysis is required to assess the
what is unknown globally about this problem. Although the existence two types of variation in the first place, and we submit that it would
of such knowledge gaps limits the generality of conclusions that can be be preferable to report the weighted and unweighted results in
drawn from the existing literature, the ability of systematic reviews and such cases.
meta-analyses to identify these gaps is a strength of these approaches Another unfortunate outcome of the high impact and growing pres-
because it directs future primary studies to the areas for which evidence is tige of meta-analysis59, coupled with the use of metrics such as citation
most needed. Other challenges for meta-analyses and systematic reviews numbers and h-indices in evaluations of research accomplishments, is an
include publication bias and research bias50, the latter describing the over- unease among some primary researchers about the fairness and rewards
or under-representation of populations, species or systems in the literature, of the scientific process8,60. Some have decried reviews as “the black-mar-
which results in a biased view of the totality. The presence of these issues ket of scientific currency”61, with calls to replace citations to reviews and
can be strongly suspected by scientists, but although their magnitude can meta-analyses with citations of primary studies61. Worse, research syn-
sometimes be estimated in a meta-analysis19,20, it cannot be truly corrected thesists in medicine have recently been described as “research parasites”62
in research syntheses55,56. Similarly, a synthesis may be constrained by of primary studies and the researchers who conduct them. On the other
either selective or incomplete data reporting in primary publications30. hand, it could be argued that primary studies without c­ ontext, compar-
One undesirable consequence of the growing recognition and high ison or summary are ultimately of limited value. Moreover, methods for
impact of meta-analysis is an increase in less-than-rigorous applications research synthesis are not the exclusive province of any one group, but

8 m a rc h 2 0 1 8 | VO L 5 5 5 | NAT U R E | 1 7 9
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.
 RESEARCH Review

can be used by primary researchers in their own areas of expertise. The and methodological developments, and is reliant on greater trans-dis-
introduction of more explicit guidelines and standards for conducting ciplinary links between statistics, computer science, the biological and
and reporting meta-analyses could address some of these grievances, and social sciences, and other scientific fields. It is not impossible to envisage
we agree that better methods for citing primary studies in meta-anal- automated systems whereby AI aids not only in the real-time acquisition
yses should be implemented to give full credit for the original studies. but also in the critical appraisal and meta-­analysis of data, potentially
‘Research parasites’ can also serve to increase scientific diversity by adding integrating different information streams to inform tailored decisions in
another ‘trophic level’, thus improving the functioning of the scientific many areas of applied science.
‘ecosystem’. The statistical methodologies that underpin and support meta-analysis
have been undergoing continual development. Areas of particular current
Advances, developments and future promise interest include multiple imputation to model missing data, advanced
Meta-analysis is the grandmother of the ‘big data’ and ‘open science’ use of meta-regression and model selection to evaluate the influence of
movements. For hundreds of years, scientists have collected data in more complex data structures and multiple covariates, and hierarchical
individual studies, based on observations and experimentation63. The modelling of multi-level data, including that from individual ‘participant’
introduction and implementation of meta-analytic techniques was the data in medicine22 and in EEC73. Network meta-analyses seek to provide
first large-scale, coordinated effort to collect and synthesize pre-existing comparisons of multiple interventions, including indirect comparisons74.
data to determine patterns, make predictions, reach generalizations and These methods are particularly useful when a set of randomized control
make evidence-based decisions. Discoveries that have resulted from the trials with pairwise comparisons of interventions has been carried out
analysis of big data, in parallel with the development of open-science with common interventions among the studies, but when not all s­ tudies
practices, transparency and the importance of replication of research, include all interventions. Developments in and applications of this
are transforming many research areas. ‘Big data’ refers to large, complex powerful approach have advanced considerably in clinical medicine over
datasets that may be mined for patterns or for making predictions, and has the past ten years75, providing better information about which treatment
been influential in a broad range of areas (for example, genomics, clima- is most effective when there are multiple treatment options and pathways.
tology and advertising). The processes involved in the searching, curation ‘Living’ reviews, which are constantly updated, can prevent stale informa-
and evaluation of data, and in quality control, are essential components tion from being cemented into belief or practice and have the p ­ otential
of big-data practice, all of which have been the subject of conceptual to change the fundamental understanding of a problem or approach,
exploration and formal methodological development in meta-analysis because knowledge is being updated and new papers are being published
for many years64. However, the approach has been different from that continuously76. Rather than summarizing information in many i­ ndividual
taken for meta-­analyses. Meta-analysis is inherently statistical, whereas reviews, living reviews and living cumulative network meta-analyses
big data has been framed within the field of computer science. Greater may also help to reduce waste in research by using the available primary
cross-­disciplinary interactions should prove productive for both fields. studies more efficiently, by identifying gaps in research and by deter-
Although formal systematic reviews and meta-analyses have long been mining when the evidence is sufficient for decision and policy making77.
established in many disciplines, they are only recently making inroads in However, their full implementation might require a reward shift both for
fields such as molecular biology and genomics. Rapid gains in scientific primary researchers and synthesists.
progress stand to be made when these methods are more fully imple- Perhaps the most important foundation for advances in meta-­analytic
mented throughout the biological sciences, and throughout science more techniques is education in high-quality research-synthesis methods.
generally. Training in meta-analytic methods and concepts should be part of the basic
Open-science practices have emphasized full and unbiased access to training for higher-degree candidates in basic and applied ­scientific fields,
scientific data65, which is of longstanding importance and central to future including research post-graduates, medical doctors and other professional
progress in meta-analysis. Pre-registration (called ‘­registration’ in some science practitioners (such as environmental ­consultants). This would
fields) of planned studies can reduce selective reporting of ­outcomes; ­formally embed their work in the context of existing ­evidence and facili-
publication of ‘registered reports’ in which the methods and proposed tate learning of both statistical and critical appraisal skills. Those involved
analyses for a study are peer-reviewed and published before the research in primary research also need a better ­understanding of meta-analysis to
is conducted can reduce publication bias. Limitations on a­ ccessing infor- exploit the revolution of open data fully. Most importantly, a new genera-
mation are serious impediments for best practices in meta-analysis. By tion of scientists, peer reviewers, editors and science-policy practitioners
minimizing selective and poor reporting and a­ dvocating full access to would benefit from an increased ­understanding of the methodologies and
the data and code associated with each analysis, open-­science standards, interpretation of evidence synthesis.
including guidelines such as those in the Equator Network (https://www. Meta-analysis can be a key tool for facilitating rapid progress in science
equator-network.org)30,66 can alleviate many ­problems in research syn- by quantifying what is known and identifying what is not yet known.
thesis and propel more rapid scientific advances. Evidence synthesis should become a regular companion to primary
In addition to the benefits that have been accrued from the increased ­scientific research to maximize the effectiveness of scientific inquiry. An
availability of unbiased information, advances in meta-analytic techniques evidence-based approach is important for progress in science, p ­ olicy,
are being driven by methodological developments. Advances include: the and medical and conservation practice. This will require collaboration
use of machine learning and artificial intelligence (AI) to screen stud- between statisticians, primary researchers and research synthesists,
ies for inclusion in systematic reviews and meta-analyses67; increasingly between meta-analysts and stakeholders, and among research synthe-
sophisticated software and models for complex meta-­regression17,47; sists across different disciplines. We are confident that, provided such
robust variance estimation in studies with small sample sizes68; meta-anal- collaborations are successful, meta-analysis will survive its ‘midlife crisis’
ysis of individual participant data; and integration of meta-analysis and and emerge stronger and with a new-found purpose.
decision support in medicine and other fields69. Bayesian meta-analysis
has been implemented in many fields and is a particularly useful approach received 4 March 2017; accepted 12 January 2018.
when external sources of information can provide valid priors70 or when
1. Jennions, M. D., Lortie, C. J. & Koricheva, J. in The Handbook of Meta-analysis in
a dataset is of sufficient quality and size that distributions can be fitted
Ecology and Evolution (eds Koricheva, J. et al.) Ch. 23, 364–380 (Princeton
to it instead of attempting to fit it to familiar distributions. Meta-analytic Univ. Press, 2013).
approaches have been used to synthesize data to address methodological 2. Roberts, P. D., Stewart, G. B. & Pullin, A. S. Are review articles a reliable source
issues such as heterogeneity and its interpretation71 and the implications of evidence to support conservation and environmental management?
A comparison with medicine. Biol. Conserv. 132, 409–423 (2006).
of the inclusion or exclusion of unpublished literature72. Better integra- 3. Bastian, H., Glasziou, P. & Chalmers, I. Seventy-five trials and eleven systematic
tion of big data, AI and meta-analysis will depend on both conceptual reviews a day: how will we ever keep up? PLoS Med. 7, e1000326 (2010).

1 8 0 | NAT U R E | VO L 5 5 5 | 8 m a rc h 2 0 1 8
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.
 Review RESEARCH

4. Borman, G. D. & Grigg, J. A. in The Handbook of Research Synthesis and 37. Cardinale, B. J. et al. Effects of biodiversity on the functioning of trophic groups
Meta-analysis 2nd edn (eds Cooper, H. M. et al.) 497–519 (Russell Sage and ecosystems. Nature 443, 989–992 (2006).
Foundation, 2009). 38. Rey Benayas, J. M., Newton, A. C., Diaz, A. & Bullock, J. M. Enhancement of
5. Ioannidis, J. P. A. The mass production of redundant, misleading, and conflicted biodiversity and ecosystem services by ecological restoration: a meta-analysis.
systematic reviews and meta-analyses. Milbank Q. 94, 485–514 (2016). Science 325, 1121–1124 (2009).
6. Koricheva, J. & Gurevitch, J. Uses and misuses of meta-analysis in plant 39. Leimu, R., Mutikainen, P. I. A., Koricheva, J. & Fischer, M. How general are
ecology. J. Ecol. 102, 828–844 (2014). positive relationships between plant population size, fitness and genetic
7. Littell, J. H. & Shlonsky, A. Making sense of meta-analysis: a critique of variation? J. Ecol. 94, 942–952 (2006).
“effectiveness of long-term psychodynamic psychotherapy”. Clin. Soc. Work J. 40. Hillebrand, H. On the generality of the latitudinal diversity gradient. Am. Nat.
39, 340–346 (2011). 163, 192–211 (2004).
8. Morrissey, M. B. Meta-analysis of magnitudes, differences and variation in 41. Gurevitch, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds
evolutionary parameters. J. Evol. Biol. 29, 1882–1904 (2016). Koricheva, J. et al.) Ch. 19, 313–320 (Princeton Univ. Press, 2013).
9. Whittaker, R. J. Meta-analyses and mega-mistakes: calling time on meta- 42. Rustad, L. et al. A meta-analysis of the response of soil respiration, net nitrogen
analysis of the species richness-productivity relationship. Ecology 91, mineralization, and aboveground plant growth to experimental ecosystem
2522–2533 (2010). warming. Oecologia 126, 543–562 (2001).
10. Begley, C. G. & Ellis, L. M. Drug development: Raise standards for preclinical 43. Adams, D. C. Phylogenetic meta-analysis. Evolution 62, 567–572 (2008).
cancer research. Nature 483, 531–533 (2012); clarification 485, 41 (2012). 44. Hadfield, J. D. & Nakagawa, S. General quantitative genetic methods for
11. Hillebrand, H. & Cardinale, B. J. A critique for meta-analyses and the comparative biology: phylogenies, taxonomies and multi-trait models for
productivity-diversity relationship. Ecology 91, 2545–2549 (2010). continuous and categorical characters. J. Evol. Biol. 23, 494–508 (2010).
12. Moher, D. et al. Preferred reporting items for systematic reviews and 45. Lajeunesse, M. J. Meta-analysis and the comparative phylogenetic method.
meta-analyses: the PRISMA statement. PLoS Med. 6, e1000097 (2009). Am. Nat. 174, 369–381 (2009).
This paper provides a consensus regarding the reporting requirements for 46. Rosenberg, M. S., Adams, D. C. & Gurevitch, J. MetaWin: Statistical Software for
medical meta-analysis and has been highly influential in ensuring good Meta-Analysis with Resampling Tests Version 1 (Sinauer Associates, 1997).
reporting practice and standardizing language in evidence-based medicine, 47. Wallace, B. C. et al. OpenMEE: intuitive, open-source software for meta-
with further guidance for protocols, individual patient data meta-analyses analysis in ecology and evolutionary biology. Methods Ecol. Evol. 8, 941–947
and animal studies. (2016).
13. Moher, D. et al. Preferred reporting items for systematic review and 48. Gurevitch, J., Morrison, J. A. & Hedges, L. V. The interaction between
meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 4, 1 (2015). competition and predation: a meta-analysis of field experiments. Am. Nat.
14. Nakagawa, S. & Santos, E. S. A. Methodological issues and advances in 155, 435–453 (2000).
biological meta-analysis. Evol. Ecol. 26, 1253–1274 (2012). 49. Adams, D. C., Gurevitch, J. & Rosenberg, M. S. Resampling tests for meta-
15. Nakagawa, S., Noble, D. W. A., Senior, A. M. & Lagisz, M. Meta-evaluation of analysis of ecological data. Ecology 78, 1277–1283 (1997).
meta-analysis: ten appraisal questions for biologists. BMC Biol. 15, 18 (2017). 50. Gurevitch, J. & Hedges, L. V. Statistical issues in ecological meta-analyses.
16. Hedges, L. & Olkin, I. Statistical Methods for Meta-analysis (Academic Press, Ecology 80, 1142–1149 (1999).
1985). 51. Schmid, C. H. & Mengersen, K. in The Handbook of Meta-analysis in Ecology and
17. Viechtbauer, W. Conducting meta-analyses in R with the metafor package. Evolution (eds Koricheva, J. et al.) Ch. 11, 145–173 (Princeton Univ. Press,
J. Stat. Softw. 36, 1–48 (2010). 2013).
18. Anzures-Cabrera, J. & Higgins, J. P. T. Graphical displays for meta-analysis: an 52. Eysenck, H. J. Exercise in mega-silliness. Am. Psychol. 33, 517 (1978).
overview with suggestions for practice. Res. Synth. Methods 1, 66–80 (2010). 53. Simberloff, D. Rejoinder to: Don’t calculate effect sizes; study ecological
19. Egger, M., Davey Smith, G., Schneider, M. & Minder, C. Bias in meta-analysis effects. Ecol. Lett. 9, 921–922 (2006).
detected by a simple, graphical test. Br. Med. J. 315, 629–634 (1997). 54. Cadotte, M. W., Mehrkens, L. R. & Menge, D. N. L. Gauging the impact of
20. Duval, S. & Tweedie, R. Trim and fill: a simple funnel-plot-based method of meta-analysis on ecology. Evol. Ecol. 26, 1153–1167 (2012).
testing and adjusting for publication bias in meta-analysis. Biometrics 56, 55. Koricheva, J., Jennions, M. D. & Lau, J. in The Handbook of Meta-analysis in
455–463 (2000). Ecology and Evolution (eds Koricheva, J. et al.) Ch. 15, 237–254 (Princeton
21. Leimu, R. & Koricheva, J. Cumulative meta-analysis: a new tool for detection of Univ. Press, 2013).
temporal trends and publication bias in ecology. Proc. R. Soc. Lond. B 271, 56. Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H. & Olkin, I. The case of the
1961–1966 (2004). misleading funnel plot. Br. Med. J. 333, 597–600 (2006).
22. Higgins, J. P. T. & Green, S. (eds) Cochrane Handbook for Systematic Reviews of 57. Vetter, D., Rucker, G. & Storch, I. Meta-analysis: a need for well-defined usage
Interventions: Version 5.1.0 (Wiley, 2011). in ecology and conservation biology. Ecosphere 4, 1–24 (2013).
This large collaborative work provides definitive guidance for the production 58. Mengersen, K., Jennions, M. D. & Schmid, C. H. in The Handbook of Meta-
of systematic reviews in medicine and is of broad interest for methods analysis in Ecology and Evolution (eds Koricheva, J. et al.) Ch. 16, 255–283
development outside the medical field. (Princeton Univ. Press, 2013).
23. Lau, J., Rothstein, H. R. & Stewart, G. B. in The Handbook of Meta-analysis in 59. Patsopoulos, N. A., Analatos, A. A. & Ioannidis, J. P. A. Relative citation impact of
Ecology and Evolution (eds Koricheva, J. et al.) Ch. 25, 407–419 (Princeton various study designs in the health sciences. J. Am. Med. Assoc. 293,
Univ. Press, 2013). 2362–2366 (2005).
24. Lortie, C. J., Stewart, G., Rothstein, H. & Lau, J. How to critically read ecological 60. Kueffer, C. et al. Fame, glory and neglect in meta-analyses. Trends Ecol. Evol.
meta-analyses. Res. Synth. Methods 6, 124–133 (2015). 26, 493–494 (2011).
25. Murad, M. H. & Montori, V. M. Synthesizing evidence: shifting the focus from 61. Cohnstaedt, L. W. & Poland, J. Review Articles: The black-market of scientific
individual studies to the body of evidence. J. Am. Med. Assoc. 309, 2217–2218 currency. Ann. Entomol. Soc. Am. 110, 90 (2017).
(2013). 62. Longo, D. L. & Drazen, J. M. Data sharing. N. Engl. J. Med. 374, 276–277
26. Rasmussen, S. A., Chu, S. Y., Kim, S. Y., Schmid, C. H. & Lau, J. Maternal obesity (2016).
and risk of neural tube defects: a meta-analysis. Am. J. Obstet. Gynecol. 198, 63. Gauch, H. G. Scientific Method in Practice (Cambridge Univ. Press, 2003).
611–619 (2008). 64. Science Staff. Dealing with data: introduction. Challenges and opportunities.
27. Littell, J. H., Campbell, M., Green, S. & Toews, B. Multisystemic therapy for Science 331, 692–693 (2011).
social, emotional, and behavioral problems in youth aged 10–17. Cochrane 65. Nosek, B. A. et al. Promoting an open research culture. Science 348,
Database Syst. Rev. https://doi.org/10.1002/14651858.CD004797.pub4 1422–1425 (2015).
(2005). 66. Stewart, L. A. et al. Preferred reporting items for a systematic review and
28. Schmidt, F. L. What do data really mean? Research findings, meta-analysis, meta-analysis of individual participant data: the PRISMA-IPD statement.
and cumulative knowledge in psychology. Am. Psychol. 47, 1173–1181 J. Am. Med. Assoc. 313, 1657–1665 (2015).
(1992). 67. Saldanha, I. J. et al. Evaluating Data Abstraction Assistant, a novel software
29. Button, K. S. et al. Power failure: why small sample size undermines the application for data abstraction during systematic reviews: protocol for a
reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013); erratum randomized controlled trial. Syst. Rev. 5, 196 (2016).
14, 451 (2013). 68. Tipton, E. & Pustejovsky, J. E. Small-sample adjustments for tests of
30. Parker, T. H. et al. Transparency in ecology and evolution: real problems, real moderators and model fit using robust variance estimation in meta-
solutions. Trends Ecol. Evol. 31, 711–719 (2016). regression. J. Educ. Behav. Stat. 40, 604–634 (2015).
31. Stewart, G. Meta-analysis in applied ecology. Biol. Lett. 6, 78–81 (2010). 69. Mengersen, K., MacNeil, M. A. & Caley, M. J. The potential for meta-analysis
32. Sutherland, W. J., Pullin, A. S., Dolman, P. M. & Knight, T. M. The need for to support decision analysis in ecology. Res. Synth. Methods 6, 111–121
evidence-based conservation. Trends Ecol. Evol. 19, 305–308 (2004). (2015).
33. Lowry, E. et al. Biological invasions: a field synopsis, systematic review, and 70. Ashby, D. Bayesian statistics in medicine: a 25 year review. Stat. Med. 25,
database of the literature. Ecol. Evol. 3, 182–196 (2013). 3589–3631 (2006).
34. Parmesan, C. & Yohe, G. A globally coherent fingerprint of climate change 71. Senior, A. M. et al. Heterogeneity in ecological and evolutionary meta-analyses:
impacts across natural systems. Nature 421, 37–42 (2003). its magnitude and implications. Ecology 97, 3293–3299 (2016).
35. Jennions, M. D., Lortie, C. J. & Koricheva, J. in The Handbook of Meta-analysis in 72. McAuley, L., Pham, B., Tugwell, P. & Moher, D. Does the inclusion of grey
Ecology and Evolution (eds Koricheva, J. et al.) Ch. 24, 381–403 (Princeton literature influence estimates of intervention effectiveness reported in
Univ. Press, 2013). meta-analyses? Lancet 356, 1228–1231 (2000).
36. Balvanera, P. et al. Quantifying the evidence for biodiversity effects on 73. Koricheva, J., Gurevitch, J. & Mengersen, K. (eds) The Handbook of Meta-
ecosystem functioning and services. Ecol. Lett. 9, 1146–1156 (2006). Analysis in Ecology and Evolution (Princeton Univ. Press, 2013).

8 m a rc h 2 0 1 8 | VO L 5 5 5 | NAT U R E | 1 8 1
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.
 RESEARCH Review

This book provides the first comprehensive guide to undertaking meta- This influential early ecological meta-analysis reports multiple experimental
analyses in ecology and evolution and is also relevant to other fields where outcomes on a longstanding and controversial topic that introduced a wide
heterogeneity is expected, incorporating explicit consideration of the range of ecologists to research synthesis methods.
different approaches used in different domains. 93. O’Rourke, K. An historical perspective on meta-analysis: dealing quantitatively
74. Lumley, T. Network meta-analysis for indirect treatment comparisons. with varying study results. J. R. Soc. Med. 100, 579–582 (2007).
Stat. Med. 21, 2313–2324 (2002). 94. Shadish, W. R. & Lecy, J. D. The meta-analytic big bang. Res. Synth. Methods 6,
75. Zarin, W. et al. Characteristics and knowledge synthesis approach for 456 246–264 (2015).
network meta-analyses: a scoping review. BMC Med. 15, 3 (2017). 95. Glass, G. V. Primary, secondary, and meta-analysis of research. Educ. Res. 5,
76. Elliott, J. H. et al. Living systematic reviews: an emerging opportunity to narrow 3–8 (1976).
the evidence-practice gap. PLoS Med. 11, e1001603 (2014). 96. DerSimonian, R. & Laird, N. Meta-analysis in clinical trials. Control. Clin. Trials 7,
77. Vandvik, P. O., Brignardello-Petersen, R. & Guyatt, G. H. Living cumulative 177–188 (1986).
network meta-analysis to reduce waste in research: a paradigmatic shift for 97. Lipsey, M. W. & Wilson, D. B. The efficacy of psychological, educational, and
systematic reviews? BMC Med. 14, 59 (2016). behavioral treatment. Confirmation from meta-analysis. Am. Psychol. 48,
78. Jarvinen, A. A meta-analytic study of the effects of female age on laying date 1181–1209 (1993).
and clutch size in the Great Tit Parus major and the Pied Flycatcher Ficedula 98. Chalmers, I. & Altman, D. G. Systematic Reviews (BMJ Publishing Group,
hypoleuca. Ibis 133, 62–67 (1991). 1995).
79. Arnqvist, G. & Wooster, D. Meta-analysis: synthesizing research findings in 99. Moher, D. et al. Improving the quality of reports of meta-analyses of
ecology and evolution. Trends Ecol. Evol. 10, 236–240 (1995). randomised controlled trials: the QUOROM statement. Quality of reporting of
80. Hedges, L. V., Gurevitch, J. & Curtis, P. S. The meta-analysis of response ratios meta-analyses. Lancet 354, 1896–1900 (1999).
in experimental ecology. Ecology 80, 1150–1156 (1999). 100. Higgins, J. P. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis.
81. Gurevitch, J., Curtis, P. S. & Jones, M. H. Meta-analysis in ecology. Adv. Ecol. Res Stat. Med. 21, 1539–1558 (2002).
32, 199–247 (2001).
82. Lajeunesse, M. J. phyloMeta: a program for phylogenetic comparative Acknowledgements We dedicate this Review to the memory of Ingram Olkin
analyses with meta-analysis. Bioinformatics 27, 2603–2604 (2011). and William Shadish, founding members of the Society for Research Synthesis
83. Pearson, K. Report on certain enteric fever inoculation statistics. Br. Med. J. 2, Methodology who made tremendous contributions to the development of
1243–1246 (1904). meta-analysis and research synthesis and to the supervision of generations of
84. Fisher, R. A. Statistical Methods for Research Workers (Oliver and Boyd, students. We thank L. Lagisz for help in preparing the figures. We are grateful
1925). to the Center for Open Science and the Laura and John Arnold Foundation for
85. Yates, F. & Cochran, W. G. The analysis of groups of experiments. J. Agric. Sci. hosting and funding a workshop, which was the origination of this article. S.N. is
28, 556–580 (1938). supported by Australian Research Council Future Fellowship (FT130100268). J.G.
86. Cochran, W. G. The combination of estimates from different experiments. acknowledges funding from the US National Science Foundation (ABI 1262402).
Biometrics 10, 101–129 (1954).
87. Smith, M. L. & Glass, G. V. Meta-analysis of psychotherapy outcome studies. Author Contributions All authors contributed equally in designing the study
Am. Psychol. 32, 752–760 (1977). and writing the manuscript, and so are listed alphabetically.
88. Glass, G. V. Meta-analysis at middle age: a personal history. Res. Synth.
Methods 6, 221–231 (2015). Author Information Reprints and permissions information is available at
89. Cooper, H. M., Hedges, L. V. & Valentine, J. C. (eds) The Handbook of Research www.nature.com/reprints. The authors declare no competing financial
Synthesis and Meta-analysis 2nd edn (Russell Sage Foundation, 2009). interests. Readers are welcome to comment on the online version of the paper.
This book is an important compilation that builds on the ground-breaking Publisher’s note: Springer Nature remains neutral with regard to jurisdictional
first edition to set the standard for best practice in meta-analysis, primarily claims in published maps and institutional affiliations. Correspondence
in the social sciences but with applications to medicine and other fields. and requests for materials should be addressed to J.G. (jessica.gurevitch@
90. Rosenthal, R. Meta-analytic Procedures for Social Research (Sage, 1991). stonybrook.edu), J.K. ([email protected]), S.N. ([email protected].
91. Hunter, J. E., Schmidt, F. L. & Jackson, G. B. Meta-analysis: Cumulating Research au) or G.S. ([email protected]).
Findings Across Studies (Sage, 1982).
92. Gurevitch, J., Morrow, L. L., Wallace, A. & Walsh, J. S. A meta-analysis of Reviewer Information Nature thanks D. Altman, M. Lajeunesse, D. Moher and
competition in field experiments. Am. Nat. 140, 539–572 (1992). G. Romero for their contribution to the peer review of this work.

1 8 2 | NAT U R E | VO L 5 5 5 | 8 m a rc h 2 0 1 8
© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved.