Research Methodology:

Meta-Analysis
S.S.S. SHASUN JAIN
COLLEGE FOR WOMEN
IQAC – RESEARCH
METHODOLOGY
WORKSHOP
RESOURCE PERSON:
Prof. (Dr.) Paresh Shah
FCMA., Ph.D., D.Ed., D.D.M., D.Ed. Psy.
Alumnus of Indian Institute of Management, Ahmedabad,
(India); Institute of Cost Accountants of India; University of
London, UK; University of Illinois, US; Darden School of
Business, University of Virginia, US; IESE Business School,
University of Navarra, Spain; University of New South
Wales, Sydney, Australia; and Alison Education, US.
This Photo by Unknown Author is licensed under CC BY
Cell no. 9824358505
EMAIL ID: [email protected]; [email protected]
Overview of the Meta-Analysis
What is a Meta-Analysis?
• Meta-Analysis vs. Narrative Review
• Meta-Analysis vs. Single Study

Seven Steps to Conduct a Meta-Analysis…

• Problem Definition
• Establishing Inclusion/Exclusion Criteria
• Literature Search and Studies Retrieval
• Coding Data and Effect Sizes for Studies
• Doing the Basic Meta-Analysis
• Moderation Analysis
• Generating the Plots (as needed)

Hands-on Exercises using Meta-Analysis Tools…

• Wilson/Lipsey SPSS Macros
• Metafor Package in R (if time permits)
• Mix 2.0 (Excel Add-in)

Qahri-Saremi & Sharma, AMCIS 2017

2
 QUESTION FIND APPRAISE SYNTHESISE

Meta-analysis
= calculated “best guess” of the true effect size
• The statistical combination of the results gives a pooled,
weighted average of the primary results
• It weights the effect size (result) of each study in relation to
sample size of the study
• Optional part of SR
Systematic reviews

Meta-analyses
Meta-analysis

This Photo by Unknown Author is licensed under CC BY-NC

What is a Meta-Analysis?
• Meta-analysis is a statistical technique for
synthesizing the results of a set of independent,
quantitative, empirical studies on a topic, in order to
determine an overall estimate of a treatment effect.

• “… now widely accepted as a method of

summarizing the results of empirical studies within
the behavioral, social and health sciences.”
Lipsey and Wilson (2001)

3
 A little bit of History…
• The Great Debate (Lipsey and Wilson, 2001)
– In 1952, Hans J. Eysenck
concluded that there were no
favourable effects of
psychotherapy, starting a raging
debate…
• 25 years of evaluation research
and hundreds of studies failed
to resolve the debate

– In 1977, Smith and Glass

statistically aggregated the
findings of 375 psychotherapy
outcome studies to show Eysenck
is wrong and psychotherapy is
indeed effective.
• In their research, they used a
method developed by Glass
(1976), called “not for want of
a less pretentious name - the
meta-analysis of research” (p.
3).
Glass, G.V. 1976. "Primary, Secondary, and
Meta-Analysis of
Research," Educational Researcher (5:10), pp. 3-
8. 4
 Meta-Analysis vs. Narrative Reviews
• Let us review this sample of studies
and answer the following two
questions:

– What tentative conclusions can you

reach about the relationship between
job satisfaction and organizational
commitment?
– Significant negative relation: 0/30.
– Significant positive relation: 19/30.
– Non-significant relation: 11/30.

– What variables do you believe can

moderate this relationship?

5
Source: Hunter & Schmidt (2004)
Reviews

This Photo by Unknown Author is licensed under CC BY

Meta-Analysis vs. Narrative
Reviews

6
Source: Hunter & Schmidt (2004)
 Meta-Analysis vs. Narrative
Reviews

• The preceding review was conducted using

standard review practices that characterize
many narrative reviews in social sciences,
including information systems.

7
 Meta-Analysis vs. Narrative
Reviews
• There is one problem, though:
– The data were constructed by a Monte Carlo simulation:
• The mean population correlation was assumed to be 0.33.
• The sample sizes were randomly chosen from a distribution
centering about 40.
– The variance across correlations and their significance levels
were determined using population correlation and the sample
size.
• The study characteristics were assigned randomly.

– In other words,
• The variation among correlations is entirely the result of sampling
error.
– The significance levels are the result of “random” sample
sizes.
• The moderator effects that appear to make sense are purely the
results of
chance.
8
 Meta-Analysis vs. Narrative
Reviews
The crucial lesson:
conflicting results in the literature may be entirely random.

– Meta-Analysis can help us differentiate between

artefactual (random) and systematic sources of variations
in results.

– Meta-analysis focuses on the direction and magnitude

of the effects across studies (effect sizes), not their
statistical significances.
• Significance testing is NOT well suited for
comparison in a review, because it is highly
dependent on sample sizes.
Sources:
Hunter & Schmidt (2004); Lindsey & Wilson (2001)

9
Meta-Analysis vs. Single Studies
• Single studies may not be sufficient to determine the
utility of an intervention or a hypothesis’ validity.
– E.g., ‘The Great Debate’ on Psychotherapy’s
effectiveness (1952-1977)
(Lipsey and Wilson, 2001).

– Single studies are limited in their statistical power,

unless the sample size is very large, which is not
common.
• Low statistical power can increase the error rates
well beyond the 5%.

10
When Can You Do Meta-analysis?
• Meta-analysis is applicable to collections of
research studies that
– Are empirical, rather than theoretical.
– Produce quantitative results, rather than qualitative
findings.
– Examine the same constructs and relationships.
– Provide findings that can be configured in a
comparable statistical form as effect sizes.

Source: Lindsey & Wilson (2001)

11
Seven Steps to Conduct a Meta-Analysis
1. Problem Definition
2. Establishing Inclusion or Exclusion Criteria
3. Literature Search and Studies Retrieval
4. Coding Data and Effect Sizes for Studies
5. Doing the Basic Meta-Analysis
– Decide on Fixed-Effects vs. Random-Effects
Meta-Analysis
– Conduct the Basic Meta-Analysis
6. Moderation Analysis
– Testing Potential Systematic Sources of Variance
Across Effect Sizes
7. Generating the Plots (as needed)
– Estimating the Publication Bias

12
Problem Definition
• Forms of Problems Suitable to a Meta-analysis (it determines
the choice of effect size):

– Association between Variables

• What is the correlation between the extent of system
use and job satisfaction?

– Central Tendency Description

• What is the mean level of satisfaction among users of a
system?

– Group Contrasts
• Are employees who are using a system more satisfied
than non-users?

– Pre-Post Contrasts
• How do employees’ job satisfaction levels change after
adopting a system?
13
Establishing Inclusion or Exclusion
Criteria:
Which Studies to Include?
• Possible components of inclusion or exclusion criteria:
– Key variables
• E.g., Job Satisfaction and Organizational Commitment.
– Distinguishing Features
• E.g., Post-adoption studies.
• Research Samples
– E.g., Studies of online banking users.
– Research methods
• E.g., Survey-based research.
– Time frame
• E.g., After 2000.
– Data Availability

• Refine the criteria as you interact with the literature.

14
Literature Search and Studies Retrieval

Search Far & Wide

(not just published studies)

• General Rule in Meta-Analysis: you should have

EVERY relevant article
– Census, not a sample.
– Look for conference presentations and proceedings

• Search in electronic reference databases.

– E.g., EBSCOhost; ScienceDirect; Web of Science;

• Search bibliography of previous reviews and pertinent

primary studies.
Literature Search and Studies Retrieval

Search Far & Wide

(not just published studies)

• Hand search journals that frequently publish on the

topic.
– AND the important journals in the field.

• Try to find “special issues” on the topic.

• Post to Listservs and relevant forums for unpublished

work.

• Contact authors who have published on your topic.

 Coding Data and Effect Sizes

What to code is an important decision…

– Effect Sizes and their related information.

• Depends on Problem Definition.
• Sample Size for the effect size.
• Reliability of Constructs used for the effect size.
– E.g., Cronbach’s Alpha.

– Potential Moderators.
• Code everything that might explain variation in findings
– Methodological Differences
» E.g., Variations in Measures and Operationalization;
Survey vs. Panel data.
– Study Characteristics
» E.g., Contextual Differences; Quality;
Characteristics of the Sample.
– Theoretical Differences Across Studies
» E.g., Firm Size; Pre-adoption vs. Post-adoption
Studies. 16
Effect Size (ES):
The Key to Meta-Analysis
• An effect size (ES) is a measure of the magnitude and
direction of a relationship between two variables or a contrast
between groups.
– It is the “dependent variable” in meta-analysis.
– The ES encodes the selected research findings on a
numeric scale.

• An ES should be:
– Comparable across studies (generally requires
standardization across studies).
– Independent of sample size.

• Different meta-analyses use different effect size indices

depending on their objectives.
– Each ES type may also have different methods of
computation.

17
Effect Size (ES):

The Correlation Coefficient

ES = r

• Represents the magnitude and direction

of association between two variables.

• Generally reported directly as “r”

(the Pearson correlation coefficient)
Source: Lindsey & Wilson (2001)

18
Effect Size (ES):
The Standardized Mean Difference
X − X G2
ES = G1 s12 (n1 −1)+ s22 (n2 −1)
spooled =
spooled n1 + n2 − 2

• Represents a standardized group contrast on a

continuous measure.

• Uses the pooled standard deviation (some

situations use control group standard
deviation).
– Cohen’s d.
– Hedges’s g.
Source: Lindsey & Wilson (2001)

19
Effect Size (ES):
The Odds-Ratio
• Represents a group contrast on a dichotomized
measure.
• The odds-ratio is based on a 2 x 2 contingency table:
– The odds of success in the treatment group relative
to the odds of success in the control group.

e.g.,
System Users
e.g.,
Job Satisfaction
ES

Frequencies
Success Failure Source: Lindsey & Wilson (2001)
Treatment Group a b
Control Group c d

20
Effect Size (ES):
The Risk Ratio
• Represents a group contrast on a dichotomized measure.

e.g., e.g.,
System Users Job Promotion

Frequencies a / (a + b)
ES =
Success Failure
c / (c + d )
Treatment Group a b
Control Group c d

• The risk ratio is also based on data from a 2 by 2

contingency table:
– The ratio of the probability of success (or failure)
for each group. Source: Lindsey & Wilson (2001)

21
Effect Size (ES):
Independent Set of Effect Sizes

• Must be dealing with an independent set of effect sizes

before proceeding with the analysis.
– One ES per a study OR
– One ES per each subsample within a study
• Remember:
– One paper can report multiple independent studies
(e.g., study1, study2)
• Therefore, one paper can provide multiple ES,
one for each independent sample.
– Multiple studies might be based on the same sample
• In this case, only one ES from that sample should be
drawn. Source: Montazemi & Qahri-Saremi (2015)

22
Effect Size (ES): Issues in Coding Effect
Sizes
• Not every article will have a nice correlation table
with sample size (N), alphas, etc.
– So, you may need to look for “r” substitutes:
• Beta values if there is only one independent
variable (IV) in the regression equation.
• Beta if this is the first step in hierarchical
regression.
• A path coefficient in SEM (if no other IVs
affecting that dependent variable).

23
 Effect Size (ES)

So far, you
should have a
list of studies
with their:
– Effect sizes
– Sample sizes
– Reliabilities
of Measures
– Potential
moderators

24
The Basic Meta-Analysis:
Fixed-Effects vs. Random-Effects Method
of Meta-Analysis
• Fixed-Effects Meta-Analysis:

– All studies are drawn from ONE homogenous

population and any variation among their effect
sizes is the result of sampling errors.
• The variance for the pooled ES is made up
only sampling errors (within study variance).
– E.g., Hedges & Olkin (1985) fixed-effects
method.

25
The Basic Meta-Analysis:
Fixed-Effects vs. Random-Effects Method of Meta-Analysis

• Random-Effects Methods of Meta-Analysis:

• Studies in a meta-analysis come from

heterogeneous populations that have different
average effect sizes, so population effect sizes can
be thought of as being sampled from a ‘super
population’:

• The variance for the pooled ES includes both

sampling error (within study variance) and
between-study variance.
– E.g., Hunter & Schmidt (2004); Hedges &
Olkin (1985) random-effects method.
26
 The Basic Meta-Analysis:
Fixed-Effects vs. Random-Effects Method of Meta-
Analysis

• Which Approach Should be Selected for the Meta-

Analysis?
– Fixed-effect models are appropriate for inferences
that extend only to the studies included in the
meta-analysis (conditional inferences).
• Limited generalization.

Source: Field & Gillette (2010)

27
The Basic Meta-Analysis:
Fixed-Effects vs. Random-Effects Method of Meta-
Analysis

– Random-effects models allow inferences that

generalize beyond the studies included in the meta-
analysis (unconditional inferences).
• When the effect sizes are not homogenous.
• Random-effects approach should be the norm in
social sciences data.
• The consequences of applying fixed-effects
methods to random-effects data can be quite
dramatic:
– Significance tests of the estimate of the
population effect size (pooled effect size) have
Type I error rates inflated from the nominal 5%
to 11% – 80%. Source: Field & Gillette (2010)

27
 The Basic Meta-Analysis:

The objective is to calculate the pooled effect size for the relation
under study, its confidence intervals, and test the homogeneity of effect
sizes in the meta-analysis:

1) Correct the effect sizes for unreliability.

2) Standardize the effect sizes using Fisher’s Z Transformation.

3) Calculate the weights for the z-transformed effect sizes.

4) Run the basic meta-analysis using the tool (calculated the pooled
effect size):
• Wilson/Lipsey SPSS Macros
• Metafor Package in R
• Mix 2.0

5) Transform the z-transformed pooled effect-size and its confidence

intervals back to correlation.

28
The Basic Meta-Analysis:
1) Correction for the sources of errors in effect sizes:
– Such as measurement errors (measures unreliability):

Corrected Correlation Observed Correlation

between variables x and y. between variables x and y.

– So if:
• rxy = .45
• rxx = .93 Reliabilities for
• ryy = .91 variables x and y.

• Then rc = 0.45 / sqrt(0.93 x 0.91) = 0.49

– No reliability reported = 1.

For other possible corrections, see:

Hunter, J.E., and Schmidt, F.L. 2004. Methods of Meta-Analysis: Correcting Error and Bias in Research
Findings, (Second ed.), Sage Publications, Inc.

32
The Basic Meta-Analysis:
1) Correction for the sources of errors in effect sizes:

33
The Basic Meta-Analysis:
2) Transform Corrected Correlations to Fisher’s Z
(standardization)
– We normalize the corrected correlations using:

 1+ r 
ES Zr = .5 ln  
 1 − r 
= 0.5 x ln (( 1 + .49)/(1 - .49))
= 0.53.

– Depending on the tools that you use, you may need to do this
transformation manually (e.g., in SPSS macros provided by
•
Wilson (2001)).
Some tools can do the transformation for you, such as metafor
package in R or Mix 2.0 software.

34
The Basic Meta-Analysis:
2) Transform Corrected Correlations to Fisher’s Z
(standardization)
1+ r 
ES Zr = .5 ln 
 1 − r 

35
The Basic Meta-Analysis:
3) Weight the effect sizes
– The standard error (SE) is a direct index of ES precision.
• The smaller the SE, the more precise the ES.
• We would like to reduce the effect of effect sizes that
are less precise (have larger SE) on our results.
• Hedges showed that the optimal weight for effect sizes is:

w= 1
SE 2

– The weight (w) for the transformed r (Zr) is simply:

• n = sample size for the pertinent effect size.
w = n-3
• Depending on the tools that you use, you may need to
do this calculation manually (e.g., in SPSS macros
provided by Wilson (2001)).

36
The Basic Meta-Analysis:

3) Weight the z-transformed effect sizes:

37
The Basic Meta-Analysis:
4) Run the basic meta-analysis using the tool
– Some of the available tools:

• Wilson/Lipsey SPSS Macros

• Metafor Package in R

• Mix 2.0

– You need to use the interface/syntaxes depending on the tool you are using.

38
The Basic Meta-Analysis:
Wilson/Lipsey SPSS Macros

Compared with 6.571, chi-square

critical value for K = 15-1 = 14.

39
The Basic Meta-Analysis:
Metafor Package in R

40
The Basic Meta-Analysis:
The pooled effect size and its confidence interval
are based on z-transformed
values, hence:
5) Transform the z-transformed pooled effect-size
and its confidence intervals back to correlation,
using:

– Depending on the tools that you use, you may

• need to do this transformation manually (e.g., in
SPSS macro provided by Wilson (2001)).
Some tools does the transformation for you,
such as metafor package in R and Mix 2.0.

41
The Basic Meta-Analysis:

42
The Basic Meta-Analysis:
Meta for Package in R

43
Your Turn:
Use Wilson/Lipsey SPSS Macros

Download the
– SPSS data file (Example_Datafile.sav),
– Wilson’s SPSS Macros,
– SPSS syntax
• from:
http://facsrv.cdm.depaul.edu/~hqahrisa/#
service

• Unzip the macros zip file

– (remember the location where the unzipped
folder is on your computer).

• Run the macros in SPSS Syntax as instructed.

44
Moderator Analysis
• Categorical Moderators
– E.g., Adoption; Gender
• Continuous
– Female Percentage (Female_Perc)

52
Moderator Analysis:
Wilson/Lipsey SPSS Macros
• For Categorical Variables using Meta Factor function (MetaF)
– Analog to a one-way ANOVA

– Useful for testing the differences across mean effect sizes for a
categorical variable.

– Example of the output for “Adoption” as a categorical moderator.

– To practice, remember to:

• Download the
– SPSS data file (Example_Datafile.sav),
– Wilson’s SPSS Macros,
– SPSS syntax
» from:
http://facsrv.cdm.depaul.edu/~hqahrisa/#service
• Unzip the macros zip file
– (remember the location where the unzipped folder is on your
computer).

53
Moderator Analysis:
Wilson/Lipsey SPSS Macros

54
Moderator Analysis:
Wilson/Lipsey SPSS Macros
• For Categorical and Continuous Variables using Meta
Regression (MetaReg)

– Analog to a regression analysis

• Estimate beta values using restricted maximum
likelihood estimator (REML).
– You can change the estimation method if you want.

– Example of the output for “Female_Perc” as a continuous

moderator.

55
Moderator Analysis:
Wilson/Lipsey SPSS Macros

56
Your Turn:
Use Wilson/Lipsey SPSS Macros

• Run the macro in SPSS Syntax as instructed:

– Run MetaF for “adoption” as a categorical moderator.

– Run MetaReg for “Female_Perc” as a continuous moderator.

57
Plots for Meta-Analysis:
Stem and Leaf from Example

Dependent Variable
Stem Leaf
.7 48
.6 00133
.5 256
.4 9
.3
.2 2
.1
.0
-.0 59
-.1 4
-.2
-.3
-.4
-.5

66
Plots for Meta-Analysis:
Stem and Leaf (Other examples)

Extraversi Neurotici Agreeable Openness Conscientio

Socio Psychological
Concession Behavior(k=16)
outcomes(k=4) on sm ness usness
Ste Lea Ste Lea Ste Leaf Ste Lea Stem Leaf
Stem Leaf Stem Leaf m f m F m m f
0.5 0.5 .5 .5 .5 .5 .5
0.4 779 0.4 .4 2 .4 .4 .4 .4 1
0.3 34 0.3 .3 0 .3 3 .3 .3 .3 1
.2 08 .2 .2 558 .2 .2
0.2 12456 0.2
8
0.1 6 0.1
.1 .1 .1 3 .1 2 .1 12
0 6 0 .0 5 .0 7 .0 .0 56 .0
0 0 -.0 1 -.0 2 -.0 2 -.0 5 -.0
0.1 -0.1
-.1 -.1 555 -.1 -.1 -.1
9
-0.2 6 -0.2 77
-.2 -.2 -.2 -.2 Source: Sharma
-.2 et al., (2013)
-0.3 -0.3 17
-.3 -.3 5 -.3 -.3 -.3
-0.4 0 0.4 -.4 -.4 -.4 -.4 -.4
-0.5 0.5 -.5 -.5 -.5 -.5 -.5

67
 FOREST PLOTS
Line of no
effect

trials

Confidence
interval

Overall
effect

Measure of
effect
QUESTION FIND APPRAISE SYNTHESISE

Smallest A. Which is the smallest

study?
B. Which is the largest
study?
C. How many are
statistically significant?
P<0.05

P<0.05
Largest

Is treatment better
than control?
How much
better?
QUESTION FIND APPRAISE SYNTHESISE

Heterogeneity

“The quality or state of being

diverse in character or content”
 QUESTION FIND APPRAISE SYNTHESISE

Heterogeneity (diversity)
• Clinical heterogeneity
Variability in the participants, interventions and/or outcomes
studied
• Methodological heterogeneity
Variability in study deign and risk of bias
• Statistical heterogeneity
The observed intervention effects being more different from each
other than we would expect due to random error (chance) alone
 QUESTION FIND APPRAISE SYNTHESISE

High heterogeneity
=
appropriate to pool data?
 QUESTION FIND APPRAISE SYNTHESISE

Are the results similar across studies? 3 tests:

1. ‘Eyeball’ test – do they look they same?
2. Formal tests
a) Proportion of variation not due to chance (I2)
• Variation between tests, now preferred estimate
• 0% to 40%: might not be important;
• 30% to 60%: may represent moderate heterogeneity;
• 50% to 90%: may represent substantial heterogeneity;
• 75% to 100%: considerable heterogeneity
b) Test of ‘Null hypothesis’ of no variation (p-value)
• Cochrane Chi-square: p<0.10 = heterogeneity
 QUESTION FIND APPRAISE SYNTHESISE

Are these trials different?

Plots for Meta-Analysis:
Forest Plot
• For showing the variability of effect sizes across studies.
– Can be used for both raw correlations as well as z-
transformed effect sizes.

68
Plots for Meta-Analysis:
Funnel Plot
• For assessing publication bias (a.k.a., file drawer problem)
– Significant findings are more likely to be published than
non-significant findings.

– Alternative measure is fail-safe N measure. (Rosenthal, 1979)

• Number of unpublished studies not included in the meta-

analysis to make the population effect size non-significant.

69
Funnel plots
Cutting Edge Issues in Meta-Analysis
• Meta-Analytic Structural Equation Modeling (MASEM)
– Using Meta-Analytic Data for testing Structural Models
using Covariance-based Structural Equation Modeling
(SEM).
• It can provide the best of both worlds: Meta-Analysis &
SEM.
• Can be used for testing hypotheses and theoretical
models.
• More complex than bivariate (basic) meta-analysis.
– Should be implemented using metaSEM package in R:
• https://courses.nus.edu.sg/course/psycwlm/internet/meta
sem/
– References:
• Cheung (2015) book on two-stage MASEM.
• Montazemi & Qahri-Saremi (2015)’s I&M paper.
70
How to do Meta-analysis in SPSS?

• https://www.discoveringstatistics.com/repository/fieldgillett
/how_to_do_a_meta_analysis.html

• A: OPEN: Data :
• Cartwright_hatton_et_al_2004.sav
• B: OPEN: Syntex
• Meta_Analysis_Basic_r.sps
• Run as shown under Menu
• Select all
• C: Output Generated : save by giving specific name as you
wish.
 References
• Cheung, M.W.-L. 2015. Meta-Analysis: A Structural Equation Modeling Approach. United Kingdom: John Wiley & Sons.
• Field, A.P. 2005. "Is the Meta-Analysis of Correlation Coefficients Accurate When Population Correlations Vary?,"
Psychological Methods (10:4), pp. 444-467.
• Field, A.P., and Gillett, R. 2010. "How to Do a Meta-Analysis," British Journal of Mathematical and Statistical Psychology
(63:3), pp. 665-694.
• Glass, G.V. 1976. "Primary, Secondary, and Meta-Analysis of Research," Educational Researcher (5:10), pp. 3-8.
• Hedges, L.V., and Olkin, I. 1985. Statistical Methods for Meta-Analysis. San Diego, CA: Academic Press.
• Huedo-Medina, T.B., Sánchez-Meca, J., Marín-Martínez, F., and Botella, J. 2006. "Assessing Heterogeneity in Meta-
Analysis: Q Statistic or I² Index?," Psychological Methods (11:2), p. 193.
• Hunter, J.E., and Schmidt, F.L. 2004. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, (Second
ed.). Sage Publications, Inc.
• Lipsey, M., and Wilson, D.B. 2001. Practical Meta-Analysis. Thousand Oaks, CA: Sage Publications:
http://mason.gmu.edu/~dwilsonb/ma.html.
• Moher D., Liberati A., Tetzlaff J., Altman D.G., The PRISMA Group. 2009. “Preferred Reporting Items for Systematic
Reviews and Meta Analyses: The PRISMA Statement”. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed1000097.
• Montazemi, A.R., and Qahri-Saremi, H. 2015. "Factors Affecting Adoption of Online Banking: A Meta-Analytic Structural
Equation Modeling Study," Information & Management (52:2), pp. 210-226.
• Rosenthal, R. 1979. "The “File Drawer Problem” and Tolerance for Null Results," Psychological Bulletin (86:3), pp. 638-
641.
• Sharma, S., Bottom, W.P., and Elfenbein, H.A. 2013. "On the Role of Personality, Cognitive Ability, and Emotional
Intelligence in Predicting Negotiation Outcomes: A Meta-Analysis," Organizational Psychology Review (3:4), pp. 293-336.
• Viechtbauer, W. 2010. "Conducting Meta-Analyses in R with the Metafor Package," Journal of Statistical Software (36:3),
pp. 1-48.

71
 Literature Search and Studies Retrieval
Preferred Reporting Items for
Systematic Reviews and Meta-
Analyses (PRISMA):

* Not Needed for

Meta-Analysis

Source: Moher D, Liberati A, Tetzlaff J, Altman

DG, The PRISMA Group (2009). Preferred
Reporting Items for Systematic Reviews and Meta
Analyses: The PRISMA Statement. PLoS Med 6(7):
e1000097. doi:10.1371/journal.pmed1000097
www.prisma-statement.org.

73
The Basic Meta-Analysis:
• Calculate the overall weighted mean effect size (fixed-effects
pooled effect size)

ES =
 (w ES) z
z
w
Remember: w = n-3.

= 1780.03 / 2888
= 0.62

– Since we have considered sampling error (SE) as the only component of variance for
weighting the effect sizes, this is the “fixed-effect” pooled effect size.

74
The Basic Meta-Analysis:
• Calculate the homogeneity index (Q statistic) for
the effect sizes.

– Q has a chi-square distribution, with number of effect sizes (k)

– 1 degrees of freedom (df).
• Q for our example is approximately 257.
– To assess existence of heterogeneity, Q should be compared
with the critical value in chi-square distribution with df = k-1
for alpha level (0.05).
• Compare with 6.571 for K = 15-1 = 14 and alpha=0.05.

– Other homogeneity indices:

Source: Huedo-Medina et al., (2006)

75
The Basic Meta-Analysis:
• Weight the effect sizes for random-effects
calculation.
– To Calculate the random-effects pooled effect size, we
should add an additional element to the weight for
the effect sizes that reflect “between-study
variance” (T2):

76
 The Basic Meta-Analysis:
• Calculate the overall weighted mean effect size (random-
effects pooled effect size) using new weights


*
(w ES)
ES = z
z
w *

Remember to use w*.

= 79.65 / 149.03
= 0.53

77
The Basic Meta-Analysis:
• Calculate the confidence interval for the random-
effects pooled effect size:

– For alpha = 0.05 (two-tailed Z (Z0.025) = 1.96):

78