t-tests and Analysis of Variance (ANOVA)

Bob McClelland

1. t-tests

1.1 t-test for Independent Samples

The t-test is the most commonly used method to evaluate the differences in means between two
groups. For example, the t-test can be used to test for a difference in test
scores between a group of patients who were given a drug and a control group who received a
placebo. Theoretically, the t-test can be used even if the sample sizes are very small (e.g., as small
as 10; some researchers claim that even smaller samples are possible), as long as the variables are
normally distributed within each group and the variation of scores in the two groups is not reliably.
The normality assumption can be evaluated by looking at the distribution of the data (via
histograms) or by performing a
normality test. The equality of variances assumption can be verified with the F-test, or you can use
the more robust Levene's test. If these conditions are not met, then you can evaluate the differences
in means between two groups using one of the nonparametric alternatives to the t- test.

The p-level reported with a t-test represents the probability of error involved in accepting a
hypothesis about the existence of a difference. Technically speaking, this is the probability of error
associated with rejecting the hypothesis of no difference between the two categories of observations
(corresponding to the groups) in the population when, in fact, the hypothesis is true. Some
researchers suggest that if the difference is in the predicted direction, you can consider only one half
(one "tail") of the probability distribution and thus divide the standard p-level reported with a t-test
(a "two-tailed" probability) by two. Others, however, suggest that you should always report the
standard, two-tailed t-test probability.

1.2 Student's t Distribution

So far, in statistics, you may have been concerned with samples taken from a normal distribution (or
large samples n > 30). In the event of the sample size (n) being small (n  30) the following procedure
would be adopted:

 The sample mean is used as the centre of the distribution

 Use the student’s t distribution instead of the normal distribution. A t-score is required
which depends on sample size and confidence level. Sample size is considered as degrees of
freedom (n - 1).

 The confidence interval is then found from the formula:

Confidence interval = x  t ( s /  n )

Therefore the value of t for a confidence level varies according to sample size.
1.3 Arrangement of Data

In order to perform the t-test for independent samples, one independent (grouping) variable (e.g.,
Gender: male/female) and at least one dependent variable (e.g., a
test score) are required. The means of the dependent variable will be compared between selected
groups based on the specified values (e.g., male and female) of the independent variable.

1.3.1 Within-group Variation

The size of a relation between two variables, such as the one measured by a difference in means
between two groups, depends to a large extent on the differentiation of values within the group.
Depending on how differentiated the values are in each group, a given "raw difference" in group
means will indicate either a stronger or weaker relationship between the independent (grouping) and
dependent variable. For example, if the mean exam mark was 62 in males and 64 in females, then
this difference of "only" 2 points would be extremely important if all values for males fell within a
range of 61 to 63, and all scores for females fell within a range of 63 to 65; for
example, we would be able to predict exam scores pretty well based on gender. However, if the
same difference of 2 was obtained from very differentiated scores (e.g., if their range was 0-90),
then we would consider the difference entirely negligible. That is to say, reduction of the within-
group variation increases the sensitivity of our test.

1.4 t-test for Dependent Samples

The t-test for dependent samples helps us to take advantage of one specific type of design in which
an important source of within-group variation (or so-called, error) can be easily identified and
excluded from the analysis. Specifically, if two groups of observations (that are to be compared) are
based on the same sample of subjects who were tested twice (e.g., before and after a treatment),
then a considerable part of the within-group variation in both groups of scores can be attributed to
the initial individual differences between subjects. Note that, in a sense, this fact is not much
different than in cases when the two groups are entirely independent (see t-test for independent
samples), where individual differences also contribute to the error variance; but in the case of
independent samples, we cannot do anything about it because we cannot identify (or "subtract") the
variation due to individual differences in subjects. However, if the same sample was tested twice,
then we can easily identify (or "subtract") this variation. Specifically, instead of treating each group
separately, and analysing raw scores, we can look only at the differences between the two measures
(e.g., "pre-test" and "post test") in each subject. By subtracting the first score from the
second for each subject and then analyzing only those "pure (paired) differences," we will exclude
the entire part of the variation in our data set that results from unequal base levels of individual
subjects. This is precisely what is being done in the t-test for dependent samples, and, as compared
to the t-test for independent samples, it always produces "better" results (i.e., it is always more
sensitive).

Below is an example of a printout from SPSS of a comparison of two factors (Factor 1 and Factor
3), in a dependent test. The hypotheses to be tested are:

Ho - There is no difference between the means of Factor 1 and Factor 3

Number of 2-tail
Variable pairs Corr Sig Mean SD SE of Mean
 --------------------------------------------------------------------------
FACTOR1 First factor identified by PO 1.3745 2.026 .126
259 .537 .000
FACTOR3 3rd Factor identified by PO a 3.1467 2.874 .179
--------------------------------------------------------------------------

Paired Differences |
Mean SD SE of Mean | t-value df 2-tail Sig
----------------------------------|-------------------------------------
-1.7722 2.473 .154 | -11.53 258 .000
95% CI (-2.075, -1.470) |

As the 2-tail significance value is given as .000 we can compare this to a significance value (in this
case 5% or 0.05). As the significance value is below 0.05 we reject Ho
at the 5% level and accept H1 concluding that there is a difference between the means of Factor 1
and Factor 3 at the 5% level.

1.5 Assumptions Made

The theoretical assumptions of the t-test for independent samples also apply to the dependent
samples test; that is, the paired differences should be normally distributed. If
these assumptions are clearly not met, then one of the nonparametric alternative tests should be
used.

2. Analysis of Variance

2.1 The Purpose of Analysis of Variance

In general, the purpose of analysis of variance (ANOVA) is to test for significant differences
between means. If we are only comparing two means, then ANOVA will give the same results as the
t-test for independent samples (if we are comparing two different groups of cases or observations),
or the t test for dependent samples (if we are comparing two variables in one set of cases or
observations). If you are not familiar with those tests you may at this point want to "brush up" on
your knowledge about those tests through readings (Kvanli, Guynes and Pavur is a good text).

The name analysis of variance may seem odd to you in that a procedure that compares means is
called analysis of variance. However, this name is derived from the fact that in order to test for
statistical significance between means, we are actually comparing (i.e., analysing) variances.

There are two areas to consider:

 Between Measures - The portion of the 'within people' (within cases) variation of a scale that
can be attributed to differences between the items.

 Within Groups - Variability of the observations in a group about their group mean. Sometimes
referred to as error variation.

Below is an example of a printout from SPSS of a comparison of the same two factors as observed
in the t-test above (Factor 1 and Factor 3), . The hypotheses to be tested are:

Ho - There is no difference between the means of Factor 1 and Factor 3

- - - - - O N E W A Y - - - - -
 Variable FACTOR1 First factor identified by PO as linked
By Variable FACTOR3 3rd Factor identified by PO as linked to

Analysis of Variance

Sum of Mean F F
Source D.F. Squares Squares Ratio Prob.

Between Groups 7 388.7223 55.5318 20.8053 .0000

As the significance value is given as .000 we can compare this to a significance value (in this case
5% or 0.05). As the significance value is below 0.05 we reject Ho
at the 5% level and accept H1 concluding that there is a difference with at least one of the means of
Factor 1 and Factor 3 at the 5% level.

2.3 Duncan procedure

In SPSS a multiple comparison procedure that ranks the group means is used as an aid to
interpretation. Here means are positioned from smallest to largest and the distance or number of
steps that two means are apart in this ranking, are used, in computing the range value for each
comparison. The test is based on the assumption that the larger the number of means being
compared, the more likely that significantly different comparisons will occur. In this procedure,
the probability of finding a significant difference, given that the two groups are in fact equal, is
sometimes less than, and never greater than, the specified significance level.
Variable FACTOR1 First factor identified by PO as linked
By Variable FACTOR3 3rd Factor identified by PO as linked to

Multiple Range Tests: Duncan test with significance level .05

The difference between two means is significant if

Step 2 3 4 5 6 7 8
RANGE 2.79 2.93 3.02 3.10 3.16 3.21 3.25

(*) Indicates significant differences which are shown in the lower triangle

G G G G G G G G
r r r r r r r r
p p p p p p p p

0 3 7 5 2 6 8 4
Mean FACTOR3

.0000 Grp 0
1.0000 Grp 3
1.7500 Grp 7 *
1.7838 Grp 5 *
2.0588 Grp 2 *
2.3962 Grp 6 * *
3.2353 Grp 8 * * * *
3.5000 Grp 4 * * * * * *