Calculating effect sizes based on means (independent populations)

Calculating effect sizes based on means – Pre-post and matched groups

Meta Analyses PUBH 7/8305

Chapters 4 and 5 (Borenstein et al.)

Hongmei Zhang

January 25, 2024

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

Table of contents

1 Calculating effect sizes based on means (independent

populations)

2 Calculating effect sizes based on means – Pre-post and matched

groups

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

Effect size based on means for two independent populations –

two commonly used definitions
q
−X̄2 (n1 −1)s12 +(n2 −1)s22
Cohen’s d: d = X̄S1pooled , Spooled = n1 +n2 −2 (pooled
standard deviation)
when sample size is small, d can be biased with respect to the
underlying truth
Hedge’s g : g = d(1 − 4 df3 −1 ) = d × J, unbiased estimate
df = n1 + n2 − 2 for two independent samples
small sample size correction:
g ′ = g (1 − 3/(4n − 9)), n = n1 + n2
recommended when n < 50, but common practice is to apply
the small sample adjustment
Such effect sizes are called standardized effect sizes → Why
this name?
What would be the definition of unstandardized effect sizes
then?
we will focus on standardized effect sizes
in general, this is more practically meaningful. → Why?
Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et
Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

Sample standard deviations

sample q
standard deviation of Cohen’s d:
d2
√
SEd = nn11+n
n2 + 2(n1 +n2 ) =
2
Vd
√
sample standard deviation of Hedge’s g : SEg = J 2 Vd
J = 1 − 4(df3)−1 , Vd = SEd2
for two independent groups, df = n1 + n2 − 2

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

Methods to compute d
published articles usually do not provide d directly
we need to use the provided statistis to compute back
how to compute d from a t-test statistic?
x¯1 −x¯2
definition of t-statistic: t = r
n1 +n2
s n1 n2

then d =?
What if the information given is p-value and sample sizes for
two independent groups? Can we still compute back to get d?
How?

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

Example
Two studies with the following information
sample means: X̄1 = 103, X̄2 = 100
sample standard deviations: S1 = 5.5, S2 = 4.5
sample size n1 = n2 = 50
Question: what is Cohen’s d and what is Hedge’s g ? And
what are their corresponding sample standard deviations?
Cohen’s d
X̄1 −X̄2 103−100
d= Spooled
= Spooled
q
(50−1)×S 2 +(50−1)×S 2
1 2
Spooled = 50+50−2
= 5.0249
substitute Spooled in the first equation by 5.0249, we have
d = 0.5970
next we calculate SEd
s
p n1 + n2 d2
SEd = Vd = +
n1 n2 2(n1 + n2 )
s
50 + 50 0.59702 √
= + = 0.0418 = 0.2044
50 × 50 2(50 + 50)

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

Hedge’s g

3
g = d × J = d(1 − )
4 df − 1
3
= 0.5970 × (1 − )
4 × (50 + 50 − 2) − 1
= 0.5970 ∗ 0.9923
=
0.5924
√ √ √
SEg = J 2 Vd = 0.99232 × 0.0418 = 0.0411 = 0.2028
SAS program to achieve the above calculations
What do we observe? Are g and d close? Which one will be
always smaller?
based on this observation, what is the intuitive justification
behind the definition of g ?
what happens if sample sizes are large?

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

What if the statistics are from pre- and post-treatment or

from two populations that are not independent?
−X̄2
Cohen’s d: d = X̄S1within , where Swithin denotes the sample
standard deviation representing within group variations
In the above, Swithin = √ Sdiff , where Sdiff is the sample
2(1−r )
standard deviation of the differences and r is the correlation,
e.g., between pre- and post-treatment scores or between
scores from two matched groups.
Sdiff is usually given in published studies.
√ q
d2
SEd = Vd = (1/n + 2n ) × 2 × (1 − r )
n denotes sample size

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

In the calculation of Swithin and SEd , r is needed, but this

statistic is usually not reported in published studies. Possible
solutions:
est. r from related studies or follow suggestions from expertise
perform sensitivity analyses with a range of plausible r ′ s
To compute g and SEg , the same formulas as those for
independent groups are applied with df = n − 1.

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et

Calculating effect sizes based on means (independent populations)
Calculating effect sizes based on means – Pre-post and matched groups

Example
sample means: X̄1 = 103, X̄2 = 100
sample standard deviation of the differences: Sdiff = 5.5
sample size n = 50
correlation between pre- and post-scores is r = 0.7
Cohen’s d
−X̄2
d = X̄S1within = 103−100
Swithin
Swithin = √ Sdiff =√ 5.5
= 7.1005
2×(1−r ) 2×(1−0.7)
substitute Swithin
qto the calculation of d, we have d = 0.4225
√ d2
SEd = Vd = (1/n + 2n ) × 2 × (1 − r ) =
q
2
(1/50 + 0.4225
2×50 ) × 2 × (1 − 0.7) = 0.1143

Hedge’s g – exercise

Hongmei Zhang Meta Analyses PUBH 7/8305 Chapters 4 and 5 (Borenstein et