5th Annual meeting of the Society for Research Synthesis Methodology

ABSTRACTS

Sala de Grados, School of Business, University of Cartagena, 5-7 July 2010

Monday, 5 July

Cross-disciplinary challenges

Gavin Stewart

Centre for Reviews and Dissemination, University of York, UK.

Research synthesis methods are fundamental to the design, conduct, analysis and interpretation of
scientific evidence across all disciplines. Arguably, synthesis of data has become a science in its own
right with an increasingly complex set of methodologies surrounding systematic review and meta-
analysis in particular. Here we attempt to provide a cross-disciplinary overview of the comparative
history and characteristics of research synthesis. As a starting point we consider synthesis in the fields
of medicine and social sciences with the longest history of use of meta-analysis and also the
environmental field, which has similar pressing needs to inform decision makers with the best-
available evidence.

Special session: bias adjustment

Bias adjustment in evidence synthesis

RM Turner1, DJ Spiegelhalter1,2, GCS Smith3 and SG Thompson1

1
MRC Biostatistics Unit, Cambridge, 2Statistical Laboratory, University of Cambridge, 3Department
of Obstetrics and Gynaecology, University of Cambridge, UK

Policy decisions often require synthesis of evidence from multiple sources, and the source studies
typically vary in rigour and in relevance to the target question. Rigour (or internal bias) reflects how
well a study estimates its intended parameters, and varies according to use of randomisation, degree of
blinding and attrition levels. Relevance (or external bias) reflects how similar the source study design
is to the target setting, with respect to study population, outcomes and interventions. We present
methods for allowing for internal and external biases in evidence synthesis.
The methods were developed in the context of a NICE technology appraisal in antenatal care, which
identified ten relevant studies. Many were historically controlled, only one was a randomised trial,
and doses, populations and outcomes varied between studies and differed from the target UK setting.
Using elicited opinion, we constructed prior distributions to represent the biases in each study, and
performed a bias-adjusted meta-analysis. Our generic bias modelling approach allows decisions to be
based on all available evidence, with less rigorous or less relevant evidence discounted using
computationally simple methods.
In further work, the bias adjustment methods have also been adapted to meta-analyses of longitudinal
observational studies. Application of the modified methods is illustrated within a systematic review

SRSM Cartagena 2010 Abstracts 1

including six studies of the relationship between objectively measured physical activity and
subsequent weight gain.

Models for potentially biased evidence in meta-analysis using empirically based priors

Nicky J Welton

Department of Community Based Medicine, University of Bristol, UK

We present methods for the combined analysis of evidence from randomized controlled trials
categorized as being at either low or high risk of bias due to a flaw in their conduct. We formulate a
bias model that incorporates between-study and between-meta-analysis heterogeneity in bias, and
uncertainty in overall mean bias. The parameters of the bias model can be estimated from collections
of previously published meta-analyses (meta-epidemiological studies). We illustrate the methods
using an illustrative example meta-analysis of clozapine in the treatment of schizophrenia. A
sensitivity analysis shows that the gain in precision from including studies at high risk of bias is likely
to be low, however numerous or large their size, and that little is gained by incorporating such studies,
unless the information from studies at low risk of bias is limited. The use of meta-epidemiological
data to inform bias parameters requires strong exchangeability assumptions, and we consider the
potential of estimating bias parameters within a mixed treatment comparison evidence structure to
avoid making such strong assumptions. We discuss approaches that might increase the value of
including studies at high risk of bias, and the acceptability of the methods in the evaluation of health
care interventions.

Adjusting for biases in a multi-parameter epidemiological synthesis model

Tony Ades

Department of Community Based Medicine, University of Bristol, UK

In multi-parameter synthesis applications there may often be data on more functions of parameters
than there are parameters. This creates the possibility of conflict between data sources. If conflict
exists under a particular model, this might either indicate that the model is mis-specified, or it might
suggest that one or more data sources may be "biased", in the sense that they are not estimating their
target parameter.
On the other hand, because there is more data than there are parameters, the size of the bias can in
principle be estimated. However, we may not know which data sources are biased and alternative
assumptions about the locus of the bias will all yield different estimates. We look at solutions that
take account of the uncertainty in which data is biassed. The presentation is illustrated with a non-
linear, 9-parameter synthesis model of the prevalence and distribution of HIV (Ades and Cliffe,
2002), based on routine surveillance and survey data.

SRSM Cartagena 2010 Abstracts 2

A case study in sensitivity to bias adjustments in a meta-analysis incorporated into a cost-
effectiveness model

Hayley Jones1, Sylwia Bujkiewicz, Rebecca Turner, Monica Lai, Nicola Cooper, Neil Hawkins, Hazel
Pilgrim, Keith Abrams, David Spiegelhalter, Alex Sutton
1
Department of Community Based Medicine, University of Bristol, UK

We continue with the example of routine antenatal anti-D prophylaxis for RhD-negative
women, introduced by Rebecca Turner earlier in this session. In Turner et al (JRSS A 2009;
172:21-47) a meta-analysis of efficacy data was performed which was adjusted for various
expected biases based on expert elicitations. We now incorporate this bias-adjusted meta-
analysis into a fully probabilistic cost-effectiveness model (Pilgrim et al, Health Technol
Assess 2009; 13:1-126).
We will further introduce the “Transparent Interactive Decision Interrogator” (TIDI), an
Excel-based user interface which runs R and WinBUGS “behind the scenes” and returns
summary statistics and graphical displays back to Excel. Using this user-friendly interface,
the user can decide interactively which of the studies to include in the meta-analysis, which
types of bias to adjust for, and also the beliefs of which experts to incorporate. This allows
the user to explore sensitivity to various choices and assumptions without expertise regarding
the underlying software or model.
Finally, we briefly consider application of a meta-epidemiological based bias adjustment, as
described by Welton et al (JRSS A 2009;172:119–136), to this case study. Preliminary
information on the average bias associated with observational versus randomised studies
from the BRANDO (Bias in Randomised AND Observational studies) database is used for
this purpose.

Mapping bias issues in 30 years of biomedical research

David Chavalarias and John Ioannidis

Center for Applied Epistemology, CNRS/Ecole Polytechnique, Paris and Department of Hygiene and
Epidemiology, University of Ioannina, Greece

Many different types of bias have been described.. Some biases may tend to coexist or be associated
with specific research settings, fields, and types of studies. We aimed to map systematically the
terminology of bias across biomedical research using advanced text-mining and clustering techniques.
The evaluation of 17M items from PubMed (1958-2008) make it possible to identify 235 bias terms
and 103 other terms that appear commonly in articles dealing with bias. Forty bias terms were used in
the title or abstract of more than 100 articles each. Pseudo-inclusion clustering identified 252 clusters
of terms for the last decade. The clusters were organized into macroscopic maps that cover a
continuum of research fields. The resulting maps highlight which types of biases tend to co-occur and
may need to be considered together and what biases are commonly encountered and discussed in
specific fields. Most of the common bias terms have had continuous use over time since their
introduction, and some (in particular confounding, selection bias, response bias, and publication bias)
show increased usage through time. This systematic mapping offers a dynamic classification of biases
in biomedical investigation and related fields and can offer insights for the multifaceted aspects of
bias.

SRSM Cartagena 2010 Abstracts 3

Tuesday, 6 July

Session 2: General methodological issues

Issues in the planning of systematic reviews in food and feed safety

Julian Higgins

MRC Biostatistics Unit, Cambridge, UK

I have recently had the opportunity to work with the Assessment Methodology Unit at the European
Food Safety Authority (EFSA) in the preparation of guidance for adopting systematic review methods
in the area of food and feed safety. Whereas many of the specific methods for systematic reviews
translate reasonably well, many of our discussions revolved around preliminary considerations in
question formulation and deciding whether a systematic review was appropriate. We found relatively
little published guidance in these areas.

I will summarize our discussions and decisions about (i) breaking down 'complex' questions into
'reviewable' questions; (ii) differentiating specific types of reviewable questions; (iii) the potential use
of 'evidence mapping' as a precursor to a systematic review; and (iv) considerations for deciding
whether or not it is worthwhile embarking on a systematic review.

Comparing the performance of alternative statistical tests for moderators in mixed-effects meta-
regression models

José A López-López1, Wolfgang Viechtbauer2, Julio Sánchez-Meca1, Fulgencio Marín-Martínez1

1
Dept. Basic Psychology and Methodology, University of Murcia, Spain, 2Maastricht University, The
Netherlands

When the effect sizes in a meta-analysis are found to be heterogeneous, researchers usually examine
whether at least part of the variability between the effect size estimates can be accounted for based on
the influence of moderator variables. The models used for this purpose are usually linear regression
models allowing for residual heterogeneity between the effect sizes, so that the resulting analysis is
typically called a mixed-effects meta-regression.
In this talk, several methods for conducting mixed-effects meta-regression analyses are compared.
Specifically, seven residual heterogeneity estimators were combined with four different methods for
testing the statistical significance of the moderators included in the model: the standard, Wald-type
method, the untruncated Knapp and Hartung method, the truncated Knapp and Hartung method (as
the authors proposed on their seminal paper in 2003) and the permutation test. The 28 resulting
combinations were compared by means of a Monte Carlo simulation.
The results did not differ with respect to the residual heterogeneity estimator used. However, some
noteworthy differences were found depending on the method employed for testing the model
coefficients. Regarding the Type I error, the standard method showed inflated rejection probabilities
when the amount of residual heterogeneity was large, especially when the number of studies was
small. On the other hand, for small amounts of residual heterogeneity, the standard method showed
overly conservative rejection probabilities (i.e., below .05). The truncated Knapp and Hartung method
was also overly conservative, but essentially across all conditions. This, in turn, lead to a noticeable

SRSM Cartagena 2010 Abstracts 4

loss of statistical power. Finally, the untruncated Knapp and Hartung method and the permutation test
showed the best performance under almost all conditions. These methods also proved to be
remarkably robust to model violations, such as when the distribution underlying the residual
heterogeneity was non-normal.
This research has been funded by the Ministerio de Ciencia e Innovación (Spanish Government) and the FEDER funds
(Project nº. PSIC2009-12172).

The Reliability Generalization Meta-analytic Approach: Do Different Statistical Methods

Matter?

Julio Sánchez-Meca, José A López-López, José A López-Pina and Fulgencio Marín-Martínez

Dept. Basic Psychology and Methodology, University of Murcia, Spain

The reliability generalization (RG) approach is a new kind of meta-analysis aimed to statistically
integrate reliability coefficients obtained in different applications of the same test, in order to
determine whether scores reliability can be generalized to different participant populations, contexts
and adaptations of the test. RG studies usually calculate an average reliability coefficient, assess the
heterogeneity assumption and search for moderator variables that can explain the variability of the
coefficients. Precursors of the RG approach have not established a single preferred analytic method,
giving freedom of choice to meta-analysts. The methods for analyzing reliability coefficients usually
applied in RG studies differ among them depending on whether: (a) coefficients are or are not
transformed, existing different transformation formulae that are applied in order to normalize their
distributions and homogenize their variances, and (b) coefficients are not weighted or some weighting
method is applied (including the assumption of a fixed- or a random-effects model). By means of a
real example, we illustrate how using different statistical methods in an RG study can influence
results. Specifically, results from an RG study of the Maudsley Obsessive-Compulsive Inventory
(MOCI) are presented. The implications of our results for the RG practice are discussed.
This research has been funded by the Fundación Séneca, Murcia County, Spain (Project nº. 08650/PHCS/08).

Sample Heterogeneity and Reliability Generalization

Juan Botella

Universidad Autónoma de Madrid

The designs of the studies providing estimates of the reliability of scores from a given test vary
considerably, especially in the sampling frames. Furthermore, the variance of the scores in any study
strongly depends of the way the participants are selected for inclusion. Although this source of
variability in the estimates has been often acknowledged in studies of Reliability Generalization (RG),
it has been rarely incorporated in the statistical analyses. First, I will show the results of several
simulations that illustrate the strong effect of this artifact in the heterogeneity of the coefficients of
internal consistency (Cronbach’s alpha). Second, I will propose a way to deal with it (Botella, Suero,
& Gambara, Psychological Methods, in press). It is based on comparing the incremental fit of nested
models, and tries to reach parsimonious conclusions. Finally, I will show several examples of how the
conclusions of a Reliability Generalization can be affected by this source of heterogeneity.

SRSM Cartagena 2010 Abstracts 5

Conducting Meta-Analyses in R with the metafor Package

Wolfgang Viechtbauer

School for Public Health and Primary Care, Maastricht University, The Netherlands

R is a computer program for performing statistical analyses and producing graphics and is becoming
the tool of choice for those conducting statistical analyses in various field. One of the great
advantages of R is that it is freely available via the internet. It is distributed with open source under
the GNU General Public License (GPL) and runs on a wide variety of platforms, including
Linux/Unix, Windows, and Mac OS X. In addition, the availability of over 2000 user-contributed add-
on packages has tremendously helped to increase R's popularity.
The metafor package (Viechtbauer, 2009) consists of a collection of functions for conducting meta-
analyses in R. The package grew out of a function written by the author several years ago
(Viechtbauer, 2006), which has since been successfully applied in several published meta-analyses.
The package allows users to easily fit fixed- and random/mixed-effects models with and without
moderators. For 2x2 table data, the Mantel-Haenszel and Peto's method are also implemented.
Moreover, the package provides various plot functions (e.g., for forest, funnel, and radial plots) and
functions for assessing the model fit, for obtaining case diagnostics, and for conducting funnel
asymmetry tests.
In this talk, I will demonstrate the current capabilities of the package with several examples, describe
some implementation details, and discuss plans for extending the package to handle multivariate and
dependent observations.
 Viechtbauer, W. (2006). MiMa: An S-Plus/R Function to Fit Meta-Analytic Mixed-, Random-, and Fixed-
Effects Models [Computer software and manual]. Retrieved from http://www.wvbauer.com/
 Viechtbauer, W. (2009). The metafor Package, Version 1.0-1 [Computer software and manual]. Retrieved
from http://cran.r-project.org/package=metafor

Session 3: Correlated estimates

Structural Equation Models in Meta-Analysis Can Control Stochastic Dependence Due to

Multiple Treatment Groups

Paul R. Hernadez, Tania B. Huedo-Medina, H. Jane Rogers, and Blair T. Johnson

University of Connecticut, Storrs, Connecticut, USA

One problem with including information from multiple treatment groups from a single study in meta-
analysis is that the effects may be correlated (i.e., stochastically dependence), especially when the
treatment groups are contrasted with a single control group (Gleser and Olkin, 2009; Kalaian and
Kasim, 2008). A small, but growing, body of methods has been proposed to address the issue of
stochastic dependence in meta-analysis due to multiple treatment groups (Becker, 2000). This study
examined how SEM can control stochastic dependence in meta-analysis (Cheung, 2008).

SRSM Cartagena 2010 Abstracts 6

The current Monte Carlo simulation study manipulated three conditions: the magnitude of the
difference between treatment and control groups (δ: 0.0 & 0.8); the number of treatment groups per
study (T: 1, 2, & 5); and sample size per group (n = 30, 100, 200).

Similar to previous studies (e.g., Raudenbush and Bryk, 2002, Van Den Noortgate and Onghena,
2003), this simulation study found dramatic biasing effects of ignoring stochastic dependence in a
univariate SEM based meta-analysis, including underestimation of the standard error (S.E.) and Type-
I error inflation. Importantly, the simulation results also indicate that the multivariate approach to
SEM based meta-analysis accurately estimated S.E. and controlled Type-I error to chance levels.
Implications of these results are discussed.

 Becker, B.J. (2000). Multivariate meta-analysis. In H.E.A. Tinsley & S.D. Brown (Eds.), Handbook of
applied multivariate statistics and mathematical modeling (pp. 499-525). Academic Press
 Cheung, M. W. (2008). A model for integrating fixed-, random-, and mixed-effects meta-analyses into
structural equation modeling. Psychological Methods, 13: 182 – 202.
 Gleser, L., J., & Olkin, I. (2009). Stochastically Dependent Effect Sizes. In H. M. Cooper, L. V. Hedges &
J. C. Valentine (Eds.), The Handbook of Research Synthesis and Meta-Analysis (2nd ed., pp. 357-376).
New York, NY: Russell Sage Foundation.
 Kalaian, S. A., & Kasim, R. M. (2008). Multilevel Methods for Meta-Analysis. In A. A. O'Connell & D. B.
McCoach (Eds.), Multilevel Modeling for Educational Data (pp. 315-343): Information Age Publishing, Inc
 Raudenbush, S. W., & Bryk, A. S. (2002). Applications in Meta-Analysis and Other Cases where Level-1
Variances are Known. In S. W. Raudenbush & A. S. Bryk (Eds.), Heirarchical Linear Models:
Applications and Data Analysis Methods (2nd ed., pp. 205-227). Thousand Oaks, CA: Sage Publications,
Inc.
 Van Den Noortgate, W., & Onghena, P. (2003). Multilevel meta-analysis: A comparison with traditional
meta-analytical procedures. Educational and Psychological Measurement, 63(5), 765-790.

Synthesis of A Partial Effect Size for the r Family

Ariel M. Aloe

University at Buffalo – SUNY, USA

The rsp index is the semi-partial correlation of a predictor with the outcome of interest. This effect
size can be computed when multiple predictor variables are included in each model in a meta-
analysis, and represents a partial effect size in the correlation family. Specifically, this index has been
proposed for use in the context of meta-analysis when primary studies report regression analyses but
do not include correlation matrices. In the current research, methods for synthesizing series of rsp
values are studied under different conditions in the primary studies. I examine variations in sample
size, the degree of correlation among predictors and between the predictors and dependent variable,
and the number of predictors in the model.

Further Results on Robust Variance Estimates for Meta-analysis Involving Correlated Effect
Size Estimates

Elizabeth Tipton, Nathan Jones

Northwestern University, USA

SRSM Cartagena 2010 Abstracts 7

Hedges, Tipton, and Johnson (2010) recently introduced a method for meta-analyzing studies with
correlated effect sizes. This method uses robust standard errors and is useful when there are multiple
outcome measures in some studies and the exact correlation structure is unknown. While the general
theory of the paper applies to all effect size measures, simulation studies and sensitivity analyses have
not been conducted for measures based on discrete outcome data. In this paper, we address the use of
robust standard errors with correlated effect sizes for the risk difference, risk ratio, and odds ratio
measures found often in medical studies. We first elucidate the underlying data generating
mechanism, then develop sensitivity analysis procedures (for varying the unknown correlation), and
provide results from simulations. Additionally, we provide an example and offer intuition regarding
the small sample properties of the estimator.

Representing multiple treatment effects using synthesized regression models

Betsy Jane Becker

Florida State University, USA

In the past I have written about ways to use synthesized correlation matrices to estimate linear
regression models in meta-analysis. Recently I have begun to examine the situation where correlations
that represent treatment effects (i.e., r values obtained by transforming standardized-mean-difference
effect sizes) are combined in a similar fashion. In this presentation I will examine synthesized
regression models that represent effects of multiple treatments derived from a single sample.
Comparisons will be made between effects based on two ways of computing the effect size d (using
the mean-square within from a two-way design versus using the standard pooled variance that would
be obtained from a t test), and the impact of confounding of (or interactions among) the treatments on
that process will also be examined.

Synthesizing evidence on multiple measures with measurement errors

G Lu & AE Ades

Department of Community Based Medicine, University of Bristol, UK ([email protected];

[email protected])

In psychological research, evidence on the treatments of interest and their comparators are often
presented on multiple outcome scales, even within the same trial. These scales often measure similar
constructs, for example the Beck and Hamilton scales for depression. It would be sensible to combine
information from the different measures to make the most efficient use of the data.
There are three main types of data: trial evidence (aggregated data on one or more outcome
measures), mapping evidence (on converting one scale-score into another) and external evidence (on
test retest, intra- inter-rater reliabilities of outcome measures and on observed correlations between
test instruments). This paper provides statistical analysis for combining these types of evidence on a
single baseline measurement. We develop a framework for synthesis of multiple outcomes that takes
account of not only correlations between outcome measures, but also measurement errors. In this
framework we ‘map’ all the outcome information into the baseline scale. The effects of measurement
error on the mappings and on the variance-covariance structures are analysed in details and then
incorporated into the synthesis process. We show that in the absence of measurement error there

SRSM Cartagena 2010 Abstracts 8

would be no benefit in combining data on different outcomes. The synthesis method is illustrated by
using data for psychological test on depression.

Session 4: Special designs

Meta-analysis of Growth Curves from Sample Means

Jack L. Vevea & Martyna Citkowicz

University of California, Merced, USA

We discuss a work in progress with emphasis on the method rather than on substantive results. A
group of former students presented us with a problem in which they wished to meta-analyze
standardized mean differences from studies with varying numbers of means that arose in the context
of repeated-measures designs. (We are deliberately vague about the details of the problem, as this is
an ongoing project on a topic of current interest in psychology, and the data are not our own.)
Although they envisioned analyzing multiple differences between means, with the number of
comparisons depending on the number of repetitions in the study, it became clear in discussion that
what they really needed was a growth curve function.
We describe an algorithm for accomplishing the analysis. First, we standardize the outcome metrics.
Next, we fit a polynomial regression for each study. Then we adjust the covariance matrix of the
sampling distribution of regression parameters to revert to the metric of raw data rather than means.
We perform a multivariate meta-analysis of the regression parameters with a random-effects error
component added to the intercept.
Note that the adjustment at the second step does not correctly reflect the true error structure of the
original repeated-measures design. As no relevant information about within-subjects error variance is
available, we conduct a sensitivity analysis by attenuating the diagonals of the covariance matrices in
varying degrees while maintaining the necessary positive definiteness of the matrix.
We illustrate the process with a partial data set.

Diagnostic accuracy reviews: should we focus on summary point(s) or summary curve(s)?

Petra Macaskill

Sydney School of Public Health, NSW, Australia

Cochrane reviews of studies of diagnostic accuracy are now being conducted. Statistical methods
currently recommended for such reviews require that a 2×2 table be extracted for each study to
provide the number of true positives, true negatives, false positives and false negatives from which an
estimate of sensitivity and also specificity of the test may be computed. Sensitivity and specificity are
expected to be negatively correlated across studies and should generally be analysed jointly.
At present, the two recommended approaches for modelling such data are (i) the bivariate model
which focuses on making inferences about a summary operating point (1-specificity, sensitivity), and

SRSM Cartagena 2010 Abstracts 9

(ii) the hierarchical summary ROC model (HSROC) of Rutter and Gatsonis that focus on making
inferences about the position and shape of a summary ROC curve. Even though the two models are
mathematically equivalent when there are no covariates, the choice of approach has implications for
how the results are reported and interpreted.
A rationale for using a particular approach will be discussed. This will be considered in the context of
exploration of heterogeneity in diagnostic test performance, and also in the context of test
comparisons.

Characteristics of Single-Case Designs Relevant to Their Synthesis.

Will Shadish & Kristynn Sullivan

University of California, Merced, USA

Single-case designs (SCDs) are short interrupted time series where an intervention is repeatedly given
to and removed from a single case (typically a person, but sometimes an aggregate like a classroom).
These designs are widely used in parts of education, psychology, and medicine when better designs
such as a randomized trial are not feasible, ethical or optimal for the patient. Despite the fact that
these designs are viewed by many researchers as providing credible evidence of cause-effect
relationships, they have not generally been included in systematic reviews. One reason for that is lack
of consensus about how data from these designs should be analyzed and aggregated. We have recently
proposed an effect size estimator that is comparable to the usual between groups standardized mean
difference statistic, and also derived a conditional variance for that estimator. The latter depends on
many features of the SCD including the autocorrelation of the data points over time, the number of
SCDs within a publication, the number of time points in the SCD and within each phase, and the
number of phases. Of particular interest in the continued development of this estimator and its
variance is their performance in computer simulations that vary the level of each of these features.
The present research will help to determine those levels by examining the existing SCD literature to
see what levels are representative of what is done when these designs are actually used. We report the
results of a survey of all publications that included SCDs during the year 2008 in a set of 21 journals
in the fields of psychology, education, and autism. We have completed initial surveys, and are in the
process now of extracting data about the design features of interest. Preliminary examination suggests
that these 21 journals published 118 articles reporting results from SCDs. Those articles contained a
total of 876 separate reports of SCDs, each report being a combination of a case and a dependent
variable. We have some additional preliminary results. For example, by far the most common metric
(80+%) for the outcome data is some form of a count, with less than 10% of the outcomes plausibly
described as normally distributed continuous data. This has significant implications for the models
used to analyze such data. We are currently coding data on additional variables, and are extracting the
raw data from the SCDs to use in computing autocorrelations. We will present as much of these data
as is available at the time of the conference.

The Visual and Narrative Interpretation of Research Syntheses

Geoffrey D. Borman & Jeffrey A. Grigg

University of Wisconsin-Madison, USA.

SRSM Cartagena 2010 Abstracts 10

Meta-analyses in the social sciences and education often provide policymakers, practitioners, and
researchers with critical new information that summarizes the central quantitative findings from a
particular research literature. However, meta-analytic articles and reports are also rather technical
pieces of work that can leave many readers without a clear sense of the results and their implications.
In this respect, both visual and narrative interpretations of research syntheses are important
considerations for presenting and describing complex results in more informative and accessible
ways.
Our work begins with a review of the typical forms of tables, graphs, and figures that researchers have
used to represent meta-analytic findings in the social sciences and education fields. We examine the
content of two prominent journals in the social sciences and education that publish research syntheses
and we contrast our findings with those of Light, Singer, and Willett (1994), who completed a similar
exercise approximately 15 years ago. Next, we briefly discuss using tables to summarize effect size
data and the characteristics of the included studies, followed by an examination of visual alternatives
for depicting meta-analytic data. We consider graphically representing effect size distributions,
representing systematic variation in effect size, and recent trends toward using figures to convey more
comprehensive accounts of the data. Unfortunately, we find that some of the most useful forms of
graphical displays are rarely or never used in prominent journals within education and the social
sciences, and we present a case for why this shortcoming should be addressed.
In this presentation, we also examine how the careful integration of quantitative and narrative
approaches to research synthesis can enhance the interpretation of meta-analyses. We outline three
key circumstances under which one should consider applying narrative techniques to quantitative
reviews. First, some of the earliest critics of meta-analysis suggested that the methodology applied a
mechanistic approach to research synthesis that sacrificed most of the information contributed by the
included studies and glossed over salient methodological and substantive differences (Eysenck, 1978;
Slavin, 1984). We believe that many contemporary applications of meta-analysis have overcome
these past criticisms through more rigorous analyses of effect size variability and modeling of the
substantive and methodological moderators that might explain variability in the observed outcomes.
However, we argue that integration of narrative forms of synthesis can help further clarify and
describe a variegated research literature. Second, there may be evidence that some literature, although
of general interest and importance, is not amenable to meta-analysis. The typical meta-analysis would
simply exclude this literature and focus only on the results that can be summarized via quantitative
techniques. We believe, though, that such a situation may call for a blending of meta-analytic and
narrative synthesis so that a more comprehensive body of research may be considered. Finally,
beyond descriptive data and significance tests of the effect sizes, we contend that researchers should
also employ narrative techniques for interpreting effect sizes, primarily in terms of understanding the
practical or policy importance of their magnitudes. In these ways and others, research syntheses that
offer the best of both meta-analysis and narrative review offer great promise.
 Eysenck, Hans J. 1978. “An Exercise in Mega-Silliness. American Psychologist, 33(5): 517.
 Light, Richard J., Judith D. Singer, and John B. Willett. 1994. “The Visual Presentation and Interpretation
of Meta-Analyses.” In The Handbook of Research Synthesis, edited by Harris Cooper and Larry V. Hedges.
New York: Russell Sage Foundation.
 Slavin, Robert E. 1984. “Meta-Analysis in Education: How Has It Been Used?” Educational Researcher,
18(8): 6-15, 24-27.

SRSM Cartagena 2010 Abstracts 11

Wednesday, 7 July

Session 5: Networks and Multilevel analyses

Structural- vs. Study-Level Characteristics in Meta-Analysis: Application to HIV Prevention

Interventions

Tania B. Huedo-Medina and Blair T. Johnson

University of Connecticut, Storrs, Connecticut, USA

Introduction Generally, primary-level studies generally omit structural-level variables due to the
difficulty to incorporating them. Yet clearly structural-level variables may interact with finer-grained
factors to influence a phenomenon (Johnson et al., in press) and meta-analytic techniques can discover
such trends (e.g., Bond & Smith, 1996). This paper illustrates these patterns by examining how geo-
temporal information (e.g., Human Development Index) may relate to the efficacy of international
HIV prevention trials, sometimes over and above the contribution of information about the trials
themselves.

Method A wide variety of systematic search strategies in different languages were used with
electronic databases to find eligible publications within two literatures. (a) One had 33 interactive,
relatively intensive controlled interventions that had condom use outcomes and took place in Latin
American or the Caribbean (N=34,597). And (b), the other had 95 interventions from around the
world that included a media component and had condom use (or other risk markers) and compared
against a control or a baseline (N=130,412). Structural-level, study, sample, and intervention
characteristics were coded. We followed a structural equation modeling strategy (Cheung, 2008) to
meta-analysis in order to accommodate variables at different levels.

Results and Conclusions Both meta-analyses revealed that overall, interventions increased condom
use. Although a number of individual-level variables significantly related to the magnitude of effect
sizes, several became non-significant or exhibited between-level interactions after structural factors
were included in the models. Specifically, interventions succeeded better in countries with lower
human development index values (an index that integrates standardized measures of measures of life
expectancy, literacy, educational attainment, and GDP per capita) , lower Gini coefficients (a measure
of income inequality), or lower HIV prevalence. Both patterns reveal that intensive HIV prevention
activities succeed best where and when the need and the inequality in the population are the greatest.
Those comparisons suggest that structural factors can be quite powerful predictors of behavior, and
may have a differential impact depending upon the cultural context. Advantages of this multi-level
approach are instant interdisciplinary implications and the possibility of geographically mapping
complex results in order to highlight best where knowledge is lacking. Disadvantages such as
restriction of range and the possibility of confounds among variables (e.g., Human Development
Index with HIV knowledge) will also be discussed.

 Bond, R. & Smith, P.B. (1996). Culture and Conformity: A Meta-analysis of studies using Asch’s (1952b,
1956) line judgment task. Psychological Bulletin, 119: 111 – 137.
 Cheung, M. W. (2008). A model for integrating fixed-, random-, and mixed-effects meta-analyses into
structural equation modeling. Psychological Methods, 13: 182 – 202.
 Johnson, B. T., Redding, C. A., DiClemente et al. (in press). A Network-Individual-Resource model for
HIV prevention. AIDS & Behavior.

SRSM Cartagena 2010 Abstracts 12

Using multilevel models for dependent effect sizes

Wim Van den Noortgate

Katholieke Universiteit Leuven, Belgium ([email protected])

Multilevel models are increasingly used to model the between-study and the sampling variation, and
look for moderator variables to explain the between-study variation. A major advantage of using
multilevel models for meta-analysis is their amazing flexibility, allowing fitting models that may
better match the kind of data and the research questions. One possibility that is seldom mentioned in
the methodological meta-analytic literature or typically is not implemented in software for meta-
analysis, is the distinction of a third level of variation to model dependencies between studies (e.g.,
occurring when several studies stem from the same research group) or within studies (e.g., occurring
when within a study multiple samples were drawn).
In the presentation we try to clarify, using real data examples and a simulation study, in which
situations three level models are appropriate to solve the problem of dependent effect sizes.

Second Order Meta-Analysis

Frank L. Schmidt, & In-Sue Oh

University of Iowa, USA & University of Alberta, Canada

This paper presents methods for second order meta-analysis. A second order meta-analysis is a meta-
analysis of a number of statistically independent meta-analyses that were conducted to estimate the
same relation in different populations. Meta-analysis greatly reduces the sampling error variance in an
estimate of an effect size or relation but does not completely eliminate sampling error. The residual
sampling error is called second order sampling error. The purpose of a second order meta-analysis is
to estimate how much of the variance in mean effect sizes across meta-analyses is attributable to
second order sampling error. We present equations and methods for second order meta-analysis for
three situations: (a) where the first order meta-analyses corrected for only sampling error; (b) where
the first order meta-analyses corrected each effect size for measurement error (and other artifacts, if
applicable); and (c) where the first order meta-analyses used the artifact distribution method to correct
for measurement error (and other artifacts if applicable). All methods and equations are random
effects (RE) models. We also present an empirical application of second order meta-analysis. For each
of five personality traits, meta-analyses have been conducted separately in five East Asian countries
relating the personality traits to job performance. For each personality trait, it appeared that the mean
correlation varied over the five countries. However, a second order meta-analysis showed that for four
of the traits all variance of these values across countries was attributable to second order sampling
error, resulting in a more parsimonious explanation. Other areas in which second order meta-analysis
might be applied are also discussed.

SRSM Cartagena 2010 Abstracts 13

Combining Pre-Post and Group Comparison Effect Sizes to Conduct a Multiple-Treatments
Meta-Analysis of Substance Abuse Treatment for Adolescents
Mark W. Lipsey, Emily Tanner-Smith, and Sandra J. Wilson

Peabody Research Institute, Vanderbilt University ([email protected])

Pre-post effect sizes for the separate arms of experimental comparisons isolate the different treatments
represented in each arm when the available studies consist mainly of treatment-treatment comparisons
but, of course, lack experimental control. Group comparison effect sizes for outcomes maintain
experimental control but do not permit the comparative effectiveness of the different treatments to be
easily determined when they are compared with each other in unsystematic and incomplete patterns.
In a meta-analysis of the effectiveness of treatment for adolescent substance abuse, we explored an
approach to integrating meta-analyses of these different effect sizes to assess comparative treatment
effectiveness. Using meta-regression to control for differences between study samples, measures, and
methods, estimates of the effects of each treatment type were derived from (a) analysis of the pre-post
effect sizes, (b) analysis of the group comparison effect sizes, and (c) analysis of synthetic group
comparisons in which one arm was statistically controlled to provide a presumptively constant basis
of comparison across the treatments in the other arm. The results of these three analyses were then
compared and integrated to draw conclusions about comparative treatment effects. This presentation
will describe that approach and invite our more sophisticated colleagues to poke holes in it.

Multiple Treatments Meta-Analysis for Categorical Outcomes

Christopher H Schmid, Thomas A Trikalinos, Ingram Olkin

Tufts Medical Center, Boston, and Stanford University, USA

Background: Meta-analyses of outcomes with three or more mutually exclusive categorical

responses (e.g., cause-specific death, other death and no death) typically employ binomial models by
collapsing or ignoring categories (e.g., death vs. no death or cause-specific death vs. no death).
Besides requiring multiple analyses, such methods may introduce bias and inefficiency by only
analyzing part of the data and ignoring correlation among the dependent responses.
Objectives: Develop a model to compare multiple treatments that form a network with a categorical
outcome. The model is applied to analyze the effect of statins on cardiovascular outcomes. The trials
form a network with four types of treatments (high and low dose statins, fibrates and controls) and
report on an outcome with six categories (fatal and non-fatal stroke, fatal and non-fatal myocardial
infarction, other causes of mortality and no event).
Methods: We apply a multinomial Bayesian model estimated using Markov chain Monte Carlo that
can incorporate missing outcomes or treatments. Such missing data may arise when some studies
report only some outcome categories or treatments and not others.
Results: We analyze data from 23 randomized trials in which each trial compares two of the four
treatments. Nine of the trials report all six outcome categories; the others report some subset, usually
because they do not split the stroke or myocardial infarction outcomes into fatal and non-fatal groups.
Using a non-informative prior distribution for the treatment effects, high dose statins reduce fatal and
non-fatal myocardial infarctions compared with control treatments. Posterior probabilities of events in
each study indicate that statins reduce the chance of poor outcomes and increase the probability of no
event.
Conclusions: We demonstrate how to estimate the relative effect of multiple treatments on multiple
categorical outcomes and produce valid simultaneous uncertainty estimates. This model should have
many applications in the clinical literature.

SRSM Cartagena 2010 Abstracts 14

Session 6: Publication bias and missing data

Quantifying Selective Reporting and the Proteus Phenomenon for Multiple Datasets with
Similar Bias

Thomas Pfeiffer, Lars Bertram and John P.A. Ioannidis

Program for Evolutionary Dynamics, Harvard University, USA and Department of Hygiene and
Epidemiology, University of Ioannina, Greece

Meta-analyses play an important role in synthesizing evidence from diverse studies and datasets that
address similar questions. One of the major obstacles for meta-analyses arises from biases in
reporting. Results published in scientific publications often present a non-random sample from all the
results that have been obtained. In particular, it is speculated that findings that do not achieve formal
statistical significance may be less likely reported than statistically significant findings. Statistical
methods have been proposed for the detection and correction of selective reporting bias. When
applied to a single meta-analysis that covers a small number of studies, however, these methods often
have limited statistical power. Here we present an extension of previous methods for analyzing
selective reporting bias. We model selective reporting based on a combined analysis of different
datasets that are assumed to be subject to the same bias. We illustrate our methods on a dataset on the
genetic basis of Alzheimer’s disease (AD). The dataset covers 1167 results from case-control studies
on 102 genetic markers. While different genetic markers may differ in their association with AD, we
assume that biases in scientific publishing are the same for all markers in the field. Analyzing such a
combined dataset increases the statistical power to quantify selective reporting and also allows
detecting more complex bias patterns. For the AD dataset we observe that initial studies on a genetic
marker tend to be substantially more biased than subsequent replications. Moreover, early replications
tend to be biased against initial findings, an observation that previously has been termed Proteus
phenomenon. Our findings imply that dynamic patters in bias, which arise from the combination of
publication bias, initial-study bias, and the Proteus phenomenon, are difficult to correct for with
conventional methods where typically simple publication bias is assumed to operate. Moreover, our
methods provide a basis for information and decision theoretical modeling of selective reporting and
thereby allow addressing the question how to optimally deal with the resulting biases.

Empirical investigation of bias in the reporting, publication, and dissemination of randomized

controlled trials in the behavioral and social sciences: A proposal

Julia H. Littell

Bryn Mawr College, USA

Background. There is ample evidence of bias in the reporting and publication of RCTs in health care.
Studies of publication bias first appeared over fifty years ago in psychology, and there is reason to
believe that this bias persists in psychology and related fields, yet there has been relatively little
research on publication bias (and related problems) outside of medicine. Many recently published
meta-analyses on psychotherapy and other social and behavioral interventions are based entirely on
published studies, and many do not make use of available methods for detecting publication bias.

SRSM Cartagena 2010 Abstracts 15

Given the lack of empirical evidence on reporting, publication, and dissemination patterns in these
fields, it is difficult to assess the validity of many published reviews.

Objectives
1. To determine the prevalence of incomplete reporting of clinical trials in psychology and social
welfare.
2. To assess the association between incomplete reporting and the direction and statistical significance
of results.
3. To assess the extent of publication bias in clinical trials in psychology and social welfare.
4. To assess the completeness and accuracy of published reviews of clinical trials.

Methods. Inception cohorts will include studies approved by research ethics committees and those
identified in prospective registers. Abstract cohorts will be drawn from presentations at conventions
of the American Psychological Association, Association for Public Policy Analysis and Management,
and the Society for Social Work and Research. We will identify the number and characteristics of
primary and secondary outcomes reported in trial protocols, conference presentations, and published
reports. Trialists will be surveyed to confirm outcomes and publication status. For a subsample of
published trials, we will track citations of these studies to identify relevant published reviews; we will
determine which results were selected for inclusion and how these results were presented in narrative
reviews, systematic reviews, and meta-analyses.

Limitations. Prospective registration of trials appears to be less common in the social and behavioral
sciences than in medicine; thus, we will probably need the cooperation of research ethics boards to
obtain a sizable inception cohort. As in previous cohort studies, we may obtain low response rate from
trialists, and their responses may be unreliable.

Benefits. Empirical evidence on reporting, publication, and dissemination patterns can inform
reviewers about steps needed to minimize bias in systematic reviews and meta-analyses in psychology
and social welfare.

The need for principles for meta-analysis utilizing non–peer-reviewed data in the public domain

Jesse A. Berlin

Johnson & Johnson Pharmaceutical Research and Development, USA

The increasing availability of clinical trial results on such websites as www.clinicaltrials.gov, has the
promise to increase the transparency of the research process, and to make conduct of systematic
reviews and meta-analyses faster and easier. This availability also raises some important questions.
For example, study reports may not always be peer-reviewed, and may not be reported in any kind of
standardized format. In this context, it’s relevant to ask whether we need additional guidelines for
meta-analyses or simply expansion on and clarification about those that exist? Two published papers
show mixed compliance with QUOROM guidelines, and particularly show that flow diagrams are
infrequently included. One of these papers (Hind and Booth) examined published monographs from
the UK NHS Health Technology Assessment (HTA) programme. (As a methodologic aside, in the
Biondi et al. paper, QUOROM scores and Guyatt-Oxman scores were not associated (R = -0.06, p =
0.86).
The presentation will provide a series of principles and questions specific to the use of publicly
“posted” clinical trial results. For example, can data and reporting standards for trials be implemented,
to allow automation of the data extraction process? Are additional reporting standards needed for

SRSM Cartagena 2010 Abstracts 16

systematic reviews that include such studies? Specific examples will be presented to highlight some
of these challenges.
Even assuming that standards can be established, and logistical obstacles can be overcome, questions
remain. What would be potential mechanisms to implement these changes? How can standards be
enforced? How do we help the public understand what level of evidence supports the conclusions that
are reported in the media?

 Biondi-Zoccai et al. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on
the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ,
doi:10.1136/bmj.38693.516782.7C (published 16 January 2006).
 Hind D, Booth A. Do health technology assessments comply with QUOROM diagram guidance? An
empirical study. BMC Medical Research Methodology 2007,7:49.

A New Approach to Meta-analysis Using Joint Partial Identification Regions

Christopher Rhoads

Northwestern University, USA

The typical approach to missing data in experimental studies has been to make untestable assumptions
about the missing data mechanism in order to obtain a point estimate of the treatment effect. As a
result, meta-analysts combining the results of these studies have almost always implicitly maintained
the same untestable assumptions about the missing data within each study. Recent research, mainly in
econometrics, has asked what can be learned about treatment effects without making untestable
assumptions about the missing data. In this approach, the results of the experiment are summarized
not by a single effect size estimate, but rather by a “partial identification region” which contains all
feasible estimates consistent with the observed data. The current paper explores methods for the
meta-analysis of partial identification regions under a fixed effects meta-analytic model. The
conditions under which the addition of more studies narrows the identification region are clarified.
The difference between the results of a typical meta-analysis that would assume missing data is
missing at random within each study and a meta-analysis of partial identification regions is illustrated
via a practical example. It is noted that in certain conditions the usual approach can result in logically
impossible estimates. Possible extensions to a random effects meta-analytic model are explored.

Using Bibliometric Temporal Trends to Predict Numbers of Studies Available for Research
Synthesis

Blair T. Johnson and Tania B. Huedo-Medina

University of Connecticut, Storrs, Connecticut, USA

Introduction Many if not most domains of investigation now have multiple systematic reviews, and
many of these are meta-analyses. Most of these reviews, in turn, attempt extensive overlapping
strategies to retrieve studies so as to maximize their evidential basis. Because most domains can be
characterized as producing new studies steadily if not steadily faster, the bibliometric trends in past
research syntheses can provide the basis for predicting the numbers for new research syntheses in
these domains, which can assist in anticipating the amount of work that is necessary to produce a

SRSM Cartagena 2010 Abstracts 17

review (e.g., Allen & Olkin, 1999). An illustrative example is provided and then trends available in
the broader domain are examined for parallels.
Method (a) Illustrative example. The numbers of studies that appeared in each year were tabulated
in a 2003 meta-analysis of HIV prevention interventions for adolescents (Johnson et al., 2003).
Number of studies was regressed on year and residuals were examined. The search was updated and
compared to the prediction. (b) More general trends. Bibliometric trends in 34 health promotion
meta-analyses summarized in Johnson et al.’s (2010) meta-synthesis were examined in parallel
fashion to determine how commonly anomalies in trends appear.
Results (a) The bibliometric trends in the 2003 literature search closely predicted the actual number
of interventions available 8 years after the close of the first search. The model predicted the numbers
of available studies better when the last two years of the search were omitted. (b) In the broader
sample, nearly all literatures exhibited increasing numbers of studies in more recent years, although
the rate of increase varied widely. Omitting the last 2 years of the trend made the slopes more positive
in 17 of the 29 meta-analyses. Meta-analyses that covered longer spans of time had more positive
slopes; those with more studies had more reliably positive slopes and appeared to omit recent studies
more often.
Conclusions Bibliometric trends can be a useful tool in planning and evaluating new research
syntheses. Rapidly expanding options for searches coupled with more refined search strategies (e.g.,
full-text searches) mean that early reviews are less likely to be predictive of future trends unless they
used extremely broad report-screening strategies. Other strengths and limitations of the approach are
discussed.

 Allen, I. E., & Olkin I. (1999). Estimating time to conduct a meta-analysis from number of citations
retrieved. JAMA, 282, 634-635.
 Johnson, B. T., Carey, M. P., Marsh, K. L., Levin, K. D., & Scott-Sheldon, L. A. J. (2003). Interventions to
reduce sexual risk for the Human Immunodeficiency Virus in adolescents, 1985-2000: A research synthesis.
Archives of Pediatrics & Adolescent Medicine, 157, 381-388.
 Johnson, B. T., Scott-Sheldon, L. A. J., & Carey, M. P. (2010). Meta-synthesis of health behavior change
meta-analyses. American Journal of Public Health. DOI: 10.2105/AJPH.2008.155200.

SRSM Cartagena 2010 Abstracts 18