NPTEL Course on

Human Computer Interaction

- An Introduction
Dr. Pradeep Yammiyavar Dr. Samit Bhattacharya
Professor, Assistant Professor,
Dept. of Design, Dept. of Computer Science
IIT Guwahati, and Engineering,
Assam, India IIT Guwahati, Assam, India

Indian Institute of Technology Guwahati

 Module 5:
Empirical Research Methods in HCI

Lecture 3:
Analysis of Empirical Data

Dr. Samit Bhattacharya

 Objective

• In the previous lectures, we learned about the

fundamentals of empirical research in the
context of HCI
– We discussed about the three themes, namely,
research question formulation, observation and
measurement and experiment design

• In this lecture, we shall focus on analysis of

empirical data
 Objective

• In particular, we shall learn about the following

– The case for statistical analysis of observed data

– Introduction to one commonly used analysis

technique, namely one-way ANOVA
Answering Empirical Questions

• Suppose, we want to determine if the text entry

speed of a text input system we proposed is
more than an existing system
• We know how to design an experiment and
observe and measure
• So, we do the following (next slide)
Answering Empirical Questions

• We conduct a user study and measure the

performance on each test condition (our system
and the existing system) over a group of
participants
• For each test condition we compute the mean
score (text entry speed) over the group of
participants
• We now have the data. What next?
Answering Empirical Questions

• We are faced with three questions

– Is there a difference?
This is obvious as we are most likely to see some
differences. However, can we conclude anything
from this difference? This brings us to the second
question
Answering Empirical Questions

• We are faced with three questions

– Is the difference large or small?
This is more difficult to answer. If we observe a
difference of, say, 30%, we can definitely say the
difference is large. However, we can’t say anything
definite about, say, a 5% difference. Clearly, the
difference figure itself can’t help us to draw any
definite conclusion. This brings us to the third
question
Answering Empirical Questions

• We are faced with three questions

– Is the difference significant or is it due to chance?
Even if the observed difference is “small”, it can still
lead us to conclude about our design if we can
determine the nature of the difference. If the
difference is found to be “significant” (not occurred
by chance), then we can say something about our
design
Answering Empirical Questions

• It is important to note that the term

“significance” is a statistical term
• The test of (statistical) significance is an
important aspect of empirical data analysis
• We can use statistical techniques for the
purpose
– The basic technique is ANOVA or ANalysis Of
VAriance
 ANOVA

• Let us go through the procedure for one-way

ANOVA
– That means, one independent variable

• Multi-way ANOVA computations are very

cumbersome to do manually
– Better to do with statistical packages
 ANOVA Illustrated

• Let’s illustrate the idea with an example

Suppose you have designed a new text entry technique
for mobile phones. You think the design is good. In fact,
you feel your method is better than the most widely used
current techniques, multi-tap and T9. You decide to
undertake some empirical research to evaluate your
invention and to compare it with the current techniques?
Suppose “better” is defined in terms of error rate
 Data

• In order to ascertain the validity of your claim,

you conducted experiments and collected the
following data (error rate of participants under
different test conditions)
Participants Your method Multi-tap T9
1 3 5 7
2 2 2 4
3 1 4 5
4 1 2 3
5 4 3 6
 ANOVA Steps - 1

• Calculate means, standard deviations (SD) and

variances for each test condition (over all
participants)

Your method Multi-tap T9

Mean 2.20 3.20 5.00
SD 1.30 1.30 1.58
Variance 1.70 1.70 2.50
 ANOVA Steps - 1

• Also calculate “grands” – values involving all

irrespective of groups
– Grand mean (mean of means) = 3.467

– Grand SD (w.r.t. grand mean) = 1.767

– Grand variance (w.r.t. grand mean) = 3.124

 ANOVA Steps - 2

• Calculate “total sum of squares (SS_T)”

SS_T = ∑ ( x _ i − mean _ grand )

2

= 43.74

x_i is the error rate value of the i-th

participant (among all)
 ANOVA Steps - 2

• An associated concept is the degrees of freedom

(DoF), which is the number of observations that
are free to vary
• DoF can be calculated simply as the (number of
things used to calculate – 1)
– For SS_T calculation, DoF = N-1
 ANOVA Steps - 3

• Next calculate the “model sum of square (SS_M)”

– Calculate (mean_group_i-mean_grand) for the i-th
group
– Square the above
– Multiply by n_i, the number of participants in the i-th
group
– Sum for all groups
 ANOVA Steps - 3

• In the example,

SS_M = 5(2.200 − 3.467)2 + 5(3.200 − 3.467)2 + 5(5.000 − 3.467)2

= 20.135

• DoF = number of group means – 1

= 3 - 1 = 2 (in our example)
 ANOVA Steps - 4

• Calculate the “residual sum of square (SS_R)” and

the corresponding DoF
SS_R = SS_T – SS_M
DoF (SS_R) = DoF (SS_T) – DoF (SS_M)

• Thus, in the example,

SS_R = 43.74 – 20.14 = 23.60
DoF (SS_R) = 14 – 2 = 12
 ANOVA Steps - 5

• Calculate two “average sum of squares” or

“mean squares (MS)”

• Model MS (MS_M) = SS_M/DoF(SS_M)

= 20.135/2 = 10.067 (for our example)

• Residue MS (MS_R) = SS_R/DOF(SS_R)

= 23.60/12 = 1.967 (for our example)
 ANOVA Steps - 6

• Calculate the “F-ratio” (simply divide MS_M by

MS_R)
– F = 10.067/1.967 = 5.12 (for our example)
• DoF associated with F-ratio are the DoFs used to
calculate the two mean squares [that is
DoF(SS_M) and DoF(SS_R)]
– In our case, these are 2, 12
• Hence, in our case, the F-ratio would be written as
F(2, 12) = 5.12
 ANOVA Steps - 6

• Look up the critical value of F

– The critical values for different “significance
levels”/thresholds (α) are available in a tabular form

– The critical values signifies the value of F that we

would expect to get by chance for α% of tests
 ANOVA Steps - 6

• Example (next slide)

– To find critical value of F(2, 12) from the table for
α=.05, look at 2nd column, 12th row for .05

– Which is 3.89

– That means, 3.89 is the F-value we would expect to

get by chance for 5% of the tests.
 Implication

• Thus, we get critical value = 3.89 for F(2,12),

α=.05

• Note that F(2, 12)=5.12 > the critical value

– Implies that the effect of test conditions has a
significant effect on the outcome w.r.t. α
 Reporting F-Statistic

• You can report the result as “my method has a

significant effect on reducing user errors
[F(2,12)=5.12, p<.05] compared to the other
methods.”
• If it is found that the effect is not significant, it is
reported as “my method has no significant effect
on reducing user errors [F(1,9)=0.634, ns]
compared to the other methods.”
 A Note of Caution

• ANOVA requires that

– Data should have normally distributed sampling
distribution and from a normally distributed
population
– Variances in each experimental condition are fairly
similar
– Observations should be independent
– Dependent variables should be measured on at least
an interval scale
 A Note of Caution

• The first two may be ignored if group sizes are

equal
– Otherwise, ALL conditions MUST have to be met