5/4/2011

SINGLE FACTOR DESIGN

Dr. Yan Liu

Department of Biomedical, Industrial and Human Factors Engineering
Wright State University

Analysis of Variance (ANOVA)

What is ANOVA
 A test of the statistical significance of the differences among the mean scores of
two or more groups on one or more independent variables
 When there is more than one dependent variable, multivariate analysis of
variance (MAOVA) is used

One-Way ANOVA
 There is one independent variable
 Each participant is randomly assigned to only one group (between-subject
design)

One-Way ANOVA for Repeated Measures
 There is one independent variable
 The same participants are used in all groups (within-subject design)

Factorial ANOVA
 There are two or more independent variables
 Each participant is randomly assigned to only one experiment condition
(between-subject factorial design)
2

1
 5/4/2011

ANOVA (Cont’d)

Factorial ANOVA for Repeated Measures
 There are two or more independent variables
 The same participants are used in all experiment conditions (within-subject
factorial design)

Factorial Mixed-Design ANOVA
 There are two or more independent variables
 Both between-subject and within-subject factors are present

Random Vs. Fixed Effects

Fixed Effects Model
 The levels of each independent variable are fixed in advance of the experiment
and we are interested in differences in responses among those specific levels
 In an experiment that compares the effectiveness of two instruction methods
(lecture vs. discussion) on students’ learning on a topic, the two levels of the
factor “type of instruction method” are fixed

Random Effects Model
 The factor levels are meant to be representative of a general population of
possible levels
 We are interested in drawing conclusions about the entire population of levels,
not just those used in the experiment design
 A researcher wants to investigate the variability in students’ learning due to
different instructors, he may choose a random sample of instructors as being
representative of the total population of instructors. The “instructor” is a random
effect

Mixed Effects Model
 Both fixed and random effects are present
4

2
 5/4/2011

Suppose you have designed a procedure that you think can help people learn and
remember things. You believe that when people use your procedure, they can learn
them faster and remember them longer.

To test your hypothesis, you compare two groups of people, an experimental group in
which individuals study your memory procedure, and a control group in which
individuals are given the same amount of time studying something irrelevant to the
task. Following the study period, all individuals are given the same set of 5 facts to
learn and then are tested the next day. You recruit 10 people, 5 of whom are randomly
assigned to the experiment group, and the rest the control group.

What type of ANOVA should be used for this experiment?

Table 1 Test Results in the Control Group

Fact total
subject 1 2 3 4 5 score
1 0 1 1 1 0 3
2 0 0 1 1 1 3
3 1 1 0 0 0 2
4 1 0 1 1 0 3
5 1 0 1 0 0 2

Table 2 Test Results in the Experimental Group

Fact total
subject 1 2 3 4 5 score
1 1 1 1 1 0 4
2 1 1 0 1 1 4
3 1 1 0 0 1 3
4 0 1 1 1 1 4
5 1 1 1 1 1 5

3
 5/4/2011

One-Way ANOVA Versus Two-Sample t-test

When two population means are compared
 Both one-way ANOVA and t-test can be used
 Unpaired two-sample t-test corresponds to one-way ANOVA with two levels
 Paired two-sample t-test corresponds to one-way ANOVA for repeated
measures with two levels

When more than two population means are compared
 It is possible to do pairwise comparisons with t-test, but conducting multiple t-
tests can lead to severe inflation of the type I error rate
 ANOVA can be used without increasing the type I error rate

One-Way ANOVA
One factor with a (a ≥ 2) levels; Compare the means of a treatments

Observations
A Totals
1 2 … n
1 y11 y12 … y1n y1.
2 y21 y22 … y2n y2.
. . . … . .
Factor A
. . . … . .

. . . … . .

a ya1 ya2 … yan ya.

Grand
y…
total

4
 5/4/2011

Two Models of One-Way ANOVA

Cell Mean Model
yij = µi + ε ij (i = 1,..., a; j = 1,..., ni )
Where yij , µi, and εij represent the jth observation in ith treatment, the mean of the
ith treatment, and the random error component that includes all other sources of
variability in the experiment, including measurement, variability arising from
uncontrolled factors, etc.

Treatment Effect Model
yij = µ + τ i + ε ij (i = 1,..., a; j = 1,..., ni )

Where µ is the overall mean, a parameter common to all treatments, and τi a parameter
unique to the ith treatment, called ith treatment effect
a

∑τ i = 0 ε ij independent N (0, σ 2 )
i =1

One-Way ANOVA with Fixed Effects Model

H0: τ1 = τ2 = …= τa = 0
H1: at least one τi ≠ 0

ANOVA Decomposition
Total sum of squares, SST, is the sum of total sum of squares due to treatments (i.e. between
treatments), SSTrt , and the sum of squares dues to error (i.e. within treatments), SSErr

SST = SSTrt + SSErr

a n
SST = ∑∑ ( yij − y.. )2 y.. : grand average
i =1 j =1
a
SSTrt = n∑ ( yi. − y.. ) 2 yi. : average of the ith treatment
i =1
a n
SSErr = ∑∑ ( yij − yi. )2 n: the number of participants in each treatment, assuming
i =1 j =1 equal sample size

10

5
 5/4/2011

One-Way ANOVA with Fixed Effects Model (Cont’d)

F-test
F = MSTrt
MSErr
~ Fa −1,n ( a −1)
Mean squares of treatment, MSTrt = SSTrt/dfTrt = SSTrt/(a-1)
Mean squares error, MSErr = SSErr/dfErr = SSErr/(a(n-1))

ANOVA Table
Sources of Sum of
df Mean squares F p-value
variation squares
Treatment a-1 SSTrt MSTrt=SSTrt/(a-1) MSTrr/MSErr
Error a(n-1) SSErr MSErr=SSErr/(a(n-1))
Total a·n-1 SSTrt

11

Self-Disclosure Example
An experiment is conducted to study the effect of physical distance on self-discourse.
Each participant is seated in a room; the interviewer comes into the room and sits at
one of three distances from the participant: close (2 ft), medium (4ft), or far (6ft).

Participants are randomly assigned to one of the three distance conditions, with 5
participants for each condition. The dependent variable is the number of personal,
revealing statements made by the participant during the interview.

Observed Self-Disclosure Statements

Total Average
Distance 1 2 3 4 5 yi. yi.
Close 33 24 31 29 34 151 30.2
Medium 21 25 19 27 26 118 23.6
Far 20 13 15 10 14 72 14.4
y.. = y..=
341 22.73
12

6
 5/4/2011

a n 2 2
SST = ∑∑ yij2 − yN.. = (332 + 242 + ... + 142 ) − 341
15
= 792.93
i =1 j =1
a 2 2
SSTreatment = 1n ∑ yi2. − yN.. = 15 (1512 + 1182 + 722 ) − 341
15 = 629.73
i =1

SSE = SST - SSTreatment = 792.93 − 629.73 = 163.2

dfTreatment = 3-1 = 2
dfE = 3(5-1) = 12
MSTreatment = 629.73/2 = 314.87
MSE = 163.2/12=13.6
Set α = 0.05, critical value: F0.05(2, 12) = 3.89, critical region: F > 3.89
F = 314.87/13.6 = 23.15 > 3.89
p-value < 0.0001
Because F is within the critical region, and p-value is much less than α =
0.05, we can reject the null hypothesis, i.e. there is evidence that the physical
distance can affect the self-disclosure 13

ANOVA Table
Sources
of Sum of Mean
variation squares df squares F p-value
Distance 629.73 2 314.87 23.15 <0.0001
Error 163.2 12 13.6
Total 792.93 14

There seem to be significant differences among the three treatment means, but
which treatment means differ significantly?

Looking at the three treatment means, we can tell that “close” and “far” would be
significantly different because they have the highest and lowest means, but we do
not know whether “close” and “medium” differ significantly or whether
“medium” and “far” differ significantly.

14

7
 5/4/2011

Multiple Comparison Tests

Fisher’s Least Significant Difference (LSD)

Tukey's Honestly Significant Difference (HSD)

Dunnett’s Test

15

Fisher’s Least Significant Difference

Fisher’s Least Significant Difference (LSD)

 Carry out all possible a(a-1)/2 pairwise t-tests

2 MSE
LSD = tα / 2 (a(n - 1)) n

where MSE is the mean square error obtained in the original ANOVA, n is the number
of samples per treatment, and a is the number of treatments

If | yi. − y j . | > LSD, we conclude that the population means µi and µj differ
significantly.

Issues with LSD

 Does not control overall type I error rate
 Not recommended in general

16

8
 5/4/2011

Self-Disclosure Example (Cont’d).

2 MSE
LSD = tα / 2 (a(n - 1)) n

α = 0.05, a = 3, n = 5, MSE = 13.6

LSD = t0.025 (12) ⋅ 2⋅13.6

5 = 2.179 ⋅ 2.33 = 5.08

Pair Difference

close vs. medium 6.6 > LSD Conclusion: All of the three
levels of distance are
close vs. far 15.8 > LSD significantly different between
each other
medium vs. far 9.2 > LSD

17

Tukey-Kramer Honestly Significant Difference

Tukey-Kramer Honestly Significant Difference (HSD)
 Probably the most commonly used multiple comparison test
 Based on the Studentized range statistic, q

For any particular pair of means among a treatments, if the larger and smaller
means are denoted as ML and MS, respectively, then Studentized range statistic can
be calculated for any particular pair as
M L −MS
q= MSE / n
~ q(a, a(n − 1))
where MSE is the mean square error obtained in the original ANOVA, n is the
number of samples per treatment

We conclude that population means µi and µj differ significantly if

| yi. − y j . |> qα (a, a (n − 1)) MS E

n
(HSD)
18

9
 5/4/2011

Self-Disclosure Example (Cont’d)

HSD = qα (a, a(n − 1)) MS E

n

α = 0.05, a = 3, n = 5, MSE = 13.6

HSD = q0.05 (3,12) ⋅ 13.6

5 = 3.77 ⋅1.65 = 6.22

Pair Difference

close vs. medium 6.6 > HSD

Conclusion: All of the three
levels of distance are
close vs. far 15.8 > HSD
significantly different between
each other
medium vs. far 9.2 > HSD

19

Dunnett’s Test

Dunnett’s Test
 Specifically designed for comparing each of the experimental groups against the
control group (not between different experimental groups)
 Based on d statistics
 For each pair of an experimental group i and the control group c, whose means
are denoted as Mi and Mc, respectively

|M i - M c |
d= MSE / n
~ dα ((a - 1), a(n - 1))
where MSE is the mean square error obtained in the original ANOVA, n
is the number of samples per treatment, and a is the number of groups
(including the control group)

We conclude that population means µi and µj differ significantly if

| yi. − y j . |> dα (a − 1, a (n − 1)) MS E

n (DD)
20

10
 5/4/2011

Self-Disclosure Example (Cont’d)

Suppose the low distance level is the “control” group, against which the
medium and far distances levels are compared

DD = dα (a − 1, a(n − 1)) MS E
n
α = 0.05, a = 3, n = 5, MSE = 13.6

DD = d0.05 (2,12) ⋅ 13.6

5 = 2.50 ⋅1.65 = 4.13

Pair Difference
Conclusion: Both medium and
close vs. medium 6.6 > DD far distances are significantly
different from the close distance
close vs. far 15.8 >DD

21

One-Way ANOVA Random Effects Model

The researcher is only interested in whether the factor has an effect instead of the
specific effect of a treatment as in the fixed effects model; therefore, the treatment
variability instead of treatment mean is to be assessed

• The same treatment effect model as in the fixed effect model

yij = µ + τ i + ε ij (i = 1,..., a; j = 1,..., ni )
a
But instead of ∑τ i = 0 , as shown in the fixed effect model, assume
i =1

τ i ~ N (0, στ2 ) , and {τi} and {εij} are independent

var(yij ) = σ τ2 + σ 2, where σ2 and σ τ are called variance components
2

2
H0: σ τ = 0
H1: σ τ2 > 0

22

11
 5/4/2011

One-Way ANOVA Random Effects Model (Cont’d)

JMP recommends using REsidual Maximum Likelihood (REML)
method for handling random effects or mixed effects model

Generates a table of RML Variance Component Estimates for random
effects

Self-Disclosure Example (Cont’d)

The variance component of Distance is quite high

23

Model Adequacy and Diagnostics

Check for Normality of Residuals
 Histogram of residuals
 Normal Q-Q plot of residuals
 Shapiro-Wilks/Kolmogorov-Smirnov test

Check for Constant Variance (Homogeneity of Variance)

 Plot of residuals vs. predicted value
 Bartlett’s or Levene’s test

Check for Independence

 Plot of residuals vs. run sequence
 Plot of residuals vs. design factor

Residual
 The residual for observation j in treatment i (yij), is defined as
eij = yij − yˆ ij yˆ ij = yi. is the estimate of yij

24

12
 5/4/2011

Self-Disclosure Example (Cont’d)

Histogram of Residual Q-Q Plot of Residual

Normality assumption is reasonable

25

Self-Disclosure Example (Cont’d)

Residual Plot (ε̂ij vs. ŷij ) Plot of Residual vs. Time

Constant variance Independence assumption is
assumption is reasonable reasonable
26

13
 5/4/2011

Remedy of Violation of Constant Variance

Violation of constant variance can affect F-test results, especially in
unbalanced experiments

Variance-Stabilizing Transformation
 In many cases when constant variance is violated, the variance is related to mean

σ Y = θ ⋅ ( µY )α ⇒ ln(σ Y ) = ln(θ ) + α ln(µY )

 We want to find a transformation on Y, Y*, that fields a constant variance σY

Y* =Yλ
It can be shown that σ Y ∝ µ λ +α −1
*

Therefore, if we set λ= 1 - α, then σ Y * is constant

27

28

14
 5/4/2011

Remedy of Violation of Constant Variance

(Cont’d)

Plot ln( S i ) over ln( yi. ), the slope of the fitting line of the plot is α
Si : sample variance of Y at the ith level of the design factor
yi. : sample mean of Y at the ith level of the design factor

29

Slope ≈ 0.5  α = 0.5 Conclusion of hypothesis testing

Therefore, λ = 0.5, i.e., y* = √y applies to the transformed data !

30

15
 5/4/2011

Remedy of Violation of Normal Distribution

Data Transformation
 Only those transformations that allow us for easy interpretation of transformed
variables are useful
 Some recommended transformations

Histogram of residuals Suggested transformation

Square root For left-skewed distribution,

first subtract all values from the
highest value plus 1, then apply
Logarithm square root, logarithm, and
inverse accordingly

Inverse

31

One-Way ANOVA with Repeated Measures

One independent variable A with a (a ≥ 2) levels; the same n participants are
in all a treatments

Data for a one-way within-subject design

Participant (S)
yi.
1 2 … n

1 y11 y12 … y1n y1.

Factor 2 y21 y22 … y2n y2.

(A) . . . … . .
. . . … . .
. . . … . .

a ya1 ya2 … yan ya.

y.j y.1 y.2 y.n y..

 The total variability in the one-way within-subject A×S design can be divided
into SSA, SSS, and SSA×S.
 Participants in the A×S design are a random sample of people from a larger
population of interest; therefore, the participants represent a random effect.
32

16
 5/4/2011

One-Way ANOVA with Repeated Measures (Cont’d)

a n a n a n
SST = ∑∑ ( yij − y.. ) 2 = n∑ ( yi. − y.. )2 + a∑ ( y. j − y.. ) 2 + ∑∑ ( yij − yi. − y. j + y.. )2
i =1 j =1 i =1 j =1 i =1 j =1

SSA SSS SSA×S

To test the hypothesis of no treatment effect of factor A

we would use F-test:

F = SSASS A /( a −1)
×S /( a −1)( n −1)
MSA
= MSA×S
~ Fa −1,( a −1)(n−1)

ANOVA table for one-way within-subject design

Sources of
df sum of squares mean squares F p-value
variation
A a-1 SSA SSA/(a-1) MSA / MSA×S
S n-1 SSS
A×S (a-1)(n-1) SSA×S SSA×S/((a-1)(n-1))

Total an-1 SST

33

Visual Interface Example

An experiment is conducted to compare users’ satisfaction with two visual
interfaces (A1 and A2). The dependent variable, user’s satisfaction, is measured
using a 10-Likert scale questionnaire. Eight participants are recruited and each
of them evaluates both interfaces.

Participant (S)
1 2 3 4 5 6 7 8 yi.
A1 6 5 5 7 4 3 5 4 4.875 y.. = 5.688
Interface A)
A2 8 6 9 6 6 5 5 7 6.5
y. j 7 5.5 7 6.5 5 4 5 5.5

34

17
 5/4/2011

Use the equation of SST, and we can get

SST = 35.44, SSA = 10.57, SSS = 15.94, SSA×B = 8.94

MSA= SSA/(a-1) =10.56/(2-1) =10.56

MSA×B= SSA×B/((a-1)(n-1)) =8.94/((2-1)(8-1) = 1.28
FA = MSA / MSA×B = 10.56/1.28 = 8.27 ~ F1,7

F0.05,1,7 = 5.59

F0.05,1,17 < FA, so we can reject the null hypothesis, i.e. there is evidence
that users’ satisfaction with the two visual interfaces differ significantly

35

Evaluation of Model Fit

Coefficient of Determination (R2)
 The proportion of the variance of the response variable that is explained by the
model

Adjusted R2
 Modification of R2 that adjusts for the number of predictor variables in a model
 Smaller than R2
Adjusted R = 1-(1- R2)*(n-1)/(n-p-1) (n: sample size, p: # predictor variables)

Root Mean Square Error (RMSE)

 Square root of the average squared difference between an actual value and its
estimate value
a n
RMSE = 1
a ⋅n ∑∑ ( y ij − yˆ ij ) 2
i =1 j =1
36

18
 5/4/2011

Determine Sample Size

Importance of Sample Size
 When the sample size is too small, it can be difficult to detect differences
between experiment conditions (a low statistical power)
 When the sample size is too large, we waste resources (both time and money)

Information Needed for Determining Sample Size

 Decide the null and alternative hypotheses and statistical test to be used

Compare a population mean to a specific value; compare the two population means,
etc.
 Decide the amount of difference to be detected, ∆
 Within-group standard deviation/variance, σerror

From previous similar studies or pilot study

Set σerror = 1.25*∆ to get started if necessary
 Significance level, α
 Power of the test, π

The probability of rejecting the null hypothesis when it is actually false

Usually between 0.50 and 0.90
37
 When you don’t have a good idea, choose π at 0.80

38

19
 5/4/2011

One-Way ANOVA Random Effects Model

The researcher is only interested in whether the factor has an effect instead of the
specific effect of a treatment as in the fixed effects model; therefore, the treatment
variability instead of treatment mean is to be assessed

• The same treatment effect model as in the fixed effect model

yij = µ + τ i + ε ij (i = 1,..., a; j = 1,..., ni )
a
But instead of ∑τ i = 0 , as shown in the fixed effect model, assume
i =1

τ i ~ N (0, στ2 ) , and {τi} and {εij} are independent

var(yij ) = σ τ2 + σ 2, where σ2 and σ τ are called variance components
2

2
H0: σ τ = 0
H1: σ τ2 > 0

39

One-Way ANOVA Random Effects Model (Cont’d)

JMP recommends using REsidual Maximum Likelihood (REML)
method for handling random effects or mixed effects model

Generates a table of RML Variance Component Estimates for random
effects

Self-Disclosure Example (Cont’d)

The variance component of Distance is quite high

40

20