Statistics for Experimental Design

Topic 4:
One-Way ANOVA:
Post-hoc Comparisons
Prior Requirements

Mohammad Darainy
 One-Way ANOVA

§ When H0 is rejected in ANOVA, then we may

want to know which group means are
different.

§ For example,
§ μ1 ≠ μ2 ≠ μ3
§ μ1 = μ2 ≠ μ3 ?
§ μ1 ≠ μ2 = μ3
 One-Way ANOVA
§ F-test gives a global effect of the
independent variable (factor) on the
dependent variable.
§ doesn’t tell which pairs of means are different.
§ We need to perform post hoc comparison
tests to determine this.
§ Post hoc (a posteriori/unplanned) comparisons
§ Decided upon after the experiment
§ Used if three or more means were compared
 Post hoc comparisons

§ We will focus on two post-hoc comparison

tests:
§ Scheffé
§ Tukey’s Honestly Significant Difference (HSD)
test
 Scheffé’s Test

§ Can be used if groups have different sizes.

§ Less sensitive to departures from the
assumptions of normality and equal
variances in the population.
§ It is the most conservative test (very unlikely
to reject H0)
 Scheffé’s Test

§ Uses F-ratio to test for a significant

difference between any two means (e.g., µ1
= µ2)
§ But, it employs a larger critical value.
§ The critical value for this test is determined
as follows:
§ Obtain the critical value of F with DF(B) = k-1
and DF(W) = N-k.
§ Multiply this value by k-1.
 Tukey’s HSD Test

§ Typically used if groups have equal sizes

and all comparisons represent simple
differences between two means.
§ This test utilizes the studentized range
statistic Q:
X j - X j'
Q=
VW / n
§ n is the equal sample size for two groups.
 Tukey’s HSD Test

§ The observed Q value is compared against

a critical value of Q (CQ) for α = 0.05 which
is associated with k and N-k .

§ Reject H0 (µj = µj’) when the observed Q

value is greater than or equal to the critical
value.
 Tukey’s HSD Test
Honestly Significant
Difference (HSD)
§ Alternatively, reject H0 if

VW
X j - X j ' ³ CQ
n
§ HSD = minimum absolute difference
between two means required for a
statistically significant difference.
 Tukey’s HSD Test

§ When sample sizes are unequal, reject H0, if:

CQ æ 1 1 ö
X j - X j' ³ VW ç + ÷
2 çN N ÷
è j j ' ø

§ This test is often called the Tukey-Kramer

test when the sample sizes are not the
same.
 Example: Tukey’s HSD Test
§ Step 1: Perform ANOVA
(k = 4, n = 10, VW = 21.175)
§ Step 2: Calculate differences in means:
Group A Group B Group C Group D
Group A - 10.28 19.17 -1.46
Group B - 8.89 -11.74
Group C - -20.63
Group D -
 Example: Tukey’s HSD Test

§ Step 3: For α = .05, CQ = 3.81.

§ Step 4: Calculate HSD:

VW
CQ = 3.81 21.175 = 5.54
n 10
 Example: Tukey’s HSD Test

Group A Group B Group C Group D

Group A - 10.28 19.17 -1.46
Group B - 8.89 -11.74
Group C - -20.63
Group D -

§ All mean differences are greater than 5.60 in

absolute value, except the difference
between Group A and Group D.
 Example: Tukey’s HSD Test

§ Thus, we may conclude that there are

statistically significant mean differences
between groups:
§ A and B;
§ A and C;
§ B and C;
§ B and D; and
§ C and D.
 Post hoc comparisons

§ Scheffé’s test is more conservative than

Tukey’s test.

§ An insignificant pair of means in Scheffé’s test

may turn out to be significant in Tukey’s test.
Increase in Type I error with multiple comparisons

§ Imagine you IV has three levels. Let's call them Group A, B

and C. So, we have in total three two by two comparisons and
assume we keep the level of type I at 𝛼 = 0.05 for each t-test.

§ For each comparison the probability of making type I error

is p=0.05 and probability of not making type I error is
q=0.95. The distribution of number of type I errors follows
binomial distribution with N=3 and
𝑃 𝑋 = 𝐶!" ×𝑝 " ×𝑞!#" , 𝑋 = [0, 𝑁]
 Increase in Type I error with multiple comparisons

§ Probability of making at lease one type I error:

§ 𝑃 𝑇𝑦𝑝𝑒 𝐼 𝑒𝑟𝑟𝑜𝑟 = 1 − .95 $ =1− 1−𝛼 $ = .1426 ≅ 3𝛼
Post hoc comparisons with JASP for Windows

§ Will the number of promotions for a product

affect the expectation of customers for the
future price of the product?

§ The more frequent promotion, the lower

expected price.
Post hoc comparisons with JASP for Windows

Pricepromotion_1.sav
Number of Expected price (USD)
promotions
1 3.78, 3.82, 4.18, …., 4.00
3 4.12, 3.91, 3.96, …., 4.11
5 3.32, 3.86, 3.96, …., 3.69
7 3.45, 3.64, 3.37, …., 3.94

§ DV = Expected price
§ Factor (IV) = Number of promotions (during the
past 10 weeks)
§ N = 160
 One-Way ANOVA with JASP
§ H0 : μ1 = μ2 = μ3 = μ4
§ H1 : Not all μ’s are equal
Descriptive Statistics
exp_price
1 3 5 7
Valid 40 40 40 40
Missing 0 0 0 0
Mean 4.224 4.063 3.759 3.549
Std. Error
0.043 0.028 0.040 0.043
of Mean
Std.
0.273 0.174 0.253 0.275
Deviation
Minimum 3.620 3.600 3.250 2.970
Maximum 4.790 4.420 4.410 4.010
 One-Way ANOVA with JASP
ANOVA - exp_price
Sum of
Cases df Mean Square F p ω²
Squares
nprom 10.989 3 3.663 59.903 < .001 0.525

Residua
9.539 156 0.061
ls

Note. Type III Sum of Squares

Means plots
 Post hoc comparisons with JASP for Windows

Post Hoc Tests

Standard
Post Hoc Comparisons - nprom
Mean Difference SE t p scheffe
1 3 0.161 0.055 2.916 0.040
5 0.465 0.055 8.410 < .001
7 0.675 0.055 12.212 < .001
3 5 0.304 0.055 5.493 < .001
7 0.514 0.055 9.296 < .001
5 7 0.210 0.055 3.802 0.003

Note. P-value adjusted for comparing a family of 4

 Post hoc comparisons with JASP for Windows

Post Hoc Tests

Standard
Post Hoc Comparisons - nprom
Mean Difference SE t p tukey
1 3 0.161 0.055 2.916 0.021
5 0.465 0.055 8.410 < .001
7 0.675 0.055 12.212 < .001
3 5 0.304 0.055 5.493 < .001
7 0.514 0.055 9.296 < .001
5 7 0.210 0.055 3.802 0.001

Note. P-value adjusted for comparing a family of 4

One-Way ANOVA Prior Requirements

§ Prior Requirements/Assumptions:
§ The distribution of observations on the
dependent variable is normal within each
group.
§ The variances of observations are equal
(homogeneity of variance).
§ Independence of observations
 Assessing Normality

§ Skewness represents the spread of the data

§ Look at descriptive statistics
§ Skewness ≈ 0
§ > 0 – positive/right skew
§ < 0 – negative/left skew
§ Look at their standard errors (SE skewness)
 Assessing Normality

§ Skewness statistic by JASP

6
§ SE skewness (≈ N
) by JASP

Skewness statistic
® t Skewness
SE Skewness
§ |tSkewnss|>3.2 violation of zero skewness
assumption.
 Assessing Normality

§ Construct histograms
§ separate histograms for each
group to assess normality
§ It doesn’t have to be perfect,
just roughly symmetric. -3 -2 -1 0 1 2 3
 Assessing Normality
§ Evaluate a normal quantile plot (or normal
probability plot).
§ Sort observations from smallest to largest.
§ Calculate the z-scores of the sorted
observations.
§ Plot the observations against the
corresponding z-scores.
§ If the data are close to normal, then the
points will lie close to some straight line.
Histogram & Normal quantile plot –
Price promotion data

Pricepromotion_1.sav
Number of Expected price (USD)
promotions
1 3.78, 3.82, 4.18, …., 4.00
3 4.12, 3.91, 3.96, …., 4.11
5 3.32, 3.86, 3.96, …., 3.69
7 3.45, 3.64, 3.37, …., 3.94

§ DV = Expected price
§ Factor (IV) = Number of promotions (during the
past 10 weeks)
§ N = 160
Histogram & Normal quantile plot –
Price promotion data
Group 1 (nprom = 1)
Histogram & Normal quantile plot –
Price promotion data
Group 2 (nprom = 3)
 Assessing Normality

§ Statistical tests of normality:

§ The Shapiro-Wilk test
§ Compares sample scores to a set of scores
generated from a normal distribution with the
sample mean and standard deviation.
§ If the test is non-significant, there is no reason to
believe the distribution is not normal.
 Assessing Normality

§ Limitation of the normality tests

§ It is very easy to give significant results
when sample size is large.
§ You can use these tests, but plot your
data as well.
 Assessing homogeneity of variance
§ Serious violation of this assumption tends to inflate
the observed value of the F statistic.
§ Too many rejections of H0 (high Type I error)
§ Levene’s test
§ Tests the null hypothesis that the population variances
are equal.
§ If Levene’s test is statistically significant (p ≤ .05), then
we may conclude that the variances are significantly
different.
§ It is very easy to have significant results when sample
size is large.
 Assessing independence of observations

§ Independent observations:
§ knowing the value of one observation gives no
clue as to that of other observation.
§ This is perhaps the most crucial assumption
for the F test.
§ It is difficult to predict how bad the F test will be.
§ There is no easy way to fix the F test when this
assumption is violated.
Assessing independence of observations

§ there is no easy test for non-independence.

§ Thus, experiments should be carefully
designed to avoid non-independent
observations.
§ Random sampling and random assignment
should be applied as much as possible.
 If these assumptions are not met…

§ A data transformation may be useful to

§ make data less skewed
§ make heterogeneous variances more
homogenous
§ Correcting positive/right skew:

Öy weak
log(y) mild
1/y strong
If these assumptions are not met…

§ If data transformation doesn’t work,

consider nonparametric tests.
Steps for conducting one-way ANOVA

§ Assure independence of observations.

§ Check normality and equal variance
assumptions
§ Create ANOVA summary table
§ If H0 is rejected, conduct multiple
comparisons for pairs of means as
necessary/desired
Example with JASP for Windows

Pricepromotion_1.sav
Number of Expected price (USD)
promotions
1 3.78, 3.82, 4.18, …., 4.00
3 4.12, 3.91, 3.96, …., 4.11
5 3.32, 3.86, 3.96, …., 3.69
7 3.45, 3.64, 3.37, …., 3.94

§ DV = Expected price
§ Factor (IV) = Number of promotions (during the
past 10 weeks)
§ N = 160
Steps for conducting one-way ANOVA

§ Assure independence of observations.

§ Descriptive statistics
§ Histograms
§ Normal quantile plots with normality tests
§ Homogeneity of variance tests
 Assessing Normality in JASP: Skewness

Descriptive Statistics
exp_price
1 3 5 7
Valid 40 40 40 40
Missing 0 0 0 0
Mean 4.224 4.063 3.759 3.549
Std. 0.273 0.174 0.253 0.275
Deviation
Skewness -0.066 -0.052 0.575 -0.366
Std. Error
of 0.374 0.374 0.374 0.374
Skewness
Minimum 3.620 3.600 3.250 2.970
Maximum 4.790 4.420 4.410 4.010
Example with JASP for Windows:
Histogram & Normal quantile plot
Group 1 (nprom = 1)
 Example with JASP for Windows:
Shapiro-Wilk test
Descriptive Statistics
exp_price
1 3 5 7
Valid 40 40 40 40
Missing 0 0 0 0
Mean 4.224 4.063 3.759 3.549
Std.
0.273 0.174 0.253 0.275
Deviation
Skewness -0.066 -0.052 0.575 -0.366
Std. Error of
0.374 0.374 0.374 0.374
Skewness
Shapiro- 0.986 0.977 0.965 0.963
Wilk
P-value of
Shapiro- 0.880 0.588 0.240 0.220
Wilk
Minimum 3.620 3.600 3.250 2.970
Maximum 4.790 4.420 4.410 4.010
Example with JASP for Windows: Test
of Homogeneity of Variance

Test for Equality of Variances (Levene's)

F df1 df2 p
3.014 3.000 156.000 0.032