Human Computer Interaction II

IT0051
 Module 5
Experimental Research
in HCI
Upon the completion of this module, the students are
expected to:

• Learn the importance of conducting experiments in real

life and relate it in the field of HCI.
• Differentiate the characteristics of experimental
research designs.
• Differentiate the different types of pre-experimental,
quasi-experimental and true-experimental research
Behavioral Learning
Using litmus paper for
testing acidity level

Freely falling bodies

EXPERIMENTS
A procedure carried out to support, refute, or validate a
hypothesis and provide insight into cause-and-effect.
Components of an Experiment
After a hypothesis is constructed, the design of an
experiment consists of three components: treatments,
units, and the assignment method.

Treatments, or conditions: the different techniques, devices, or

procedures that we want to compare

Units: the objects to which we apply the experiment treatments. In HCI

research, the units are normally human subjects with specific
characteristics, such as gender, age, or computing experience.

Assignment method: the way in which the experimental units are

assigned different treatments.
Components of an Experiment
Suppose a researcher is running an experiment to compare typing speed using a traditional QWERTY
keyboard and a DVORAK keyboard. The treatment of this experiment is the type of keyboard: QWERTY or
DVORAK. The experiment units are the participants recruited to join the study. To achieve the goal of fair
comparison, the researchers would have to require that the participants have no previous experience using
either keyboard.
Components of an Experiment
• The power of experimental research lies in its ability to uncover causal relations. The major
reason why experimental research can achieve this goal is because of complete
randomization. Randomization refers to the random assignment of treatments to the
experimental units or participants (Oehlert, 2000).

• In a totally randomized experiment, no one, including the investigators themselves, is able to

predict the condition to which a participant is going to be assigned.

• This effectively controls the influence of hidden factors and allows a clean comparison
between the experiment conditions.

• Traditional randomization methods include tossing a coin, throwing dice, spinning a roulette
wheel, or drawing capsules out of an urn. However, these types of randomization are rarely
used in behavioral research and HCI studies nowadays. One method to randomize the
selection of experimental conditions or other factors is the use of a random digit table.
An experimental group is the group that receives the variable being
tested in an experiment while the control group does not receive a
treatment.
If your hypothesis is correct, then it is a theory that could work
every single time the experiment has been performed by scientists.
Research Hypotheses

An experiment normally starts with a research

hypothesis. A hypothesis is a precise problem
statement that can be directly tested through an
empirical investigation. Compared with a theory, a
hypothesis is a smaller, more focused statement
that can be examined by a single experiment
(Rosenthal and Rosnow, 2008).

Fitts' law (Fitts, 1954), one of the most widely

accepted theories in the HCI field. It states a
general relationship between movement time,
navigation distance, and target size for pointing
tasks in an interface:
A hypothesis is a statement or an assumption about
relationships between two variables that serves as a proposed
explanation for a phenomenon.
An experiment normally has at least one null hypothesis and one alternative
hypothesis. A null hypothesis typically states that there is no difference between
experimental treatments. The alternative hypothesis is always a statement that is
mutually exclusive with the null hypothesis.
Examples:
H0: There is no difference in user satisfaction rating between the pull-down menu and the pop-
up menu. null hypothesis

H1: There is a difference in user satisfaction rating between the pull-down menu and the pop-
up menu. alternative hypothesis

H0: There is no difference between the pull-down menu and the pop-up menu in the time
spent locating pages. null hypothesis

H1: There is a difference between the pull-down menu and the pop-up menu in the time spent
locating pages. alternative hypothesis
In order to conduct a successful experiment, it is crucial to start with
one or more good hypotheses (Durbin, 2004). A good hypothesis
normally satisfies the following criteria:

• is presented in precise, lucid language;

• is focused on a problem that is testable in one experiment;
• clearly states the control groups or conditions of the experiment.
Significance Tests

A test significance is a formal procedure for comparing observed data with a

claim(also called a hypothesis), the truth of which is being assessed.
Significance Tests
Almost all experimental investigations are analyzed and reported
through significance tests.
Significance Tests
Why do you need to run significance tests on your data?
Internal Validity
is the extent to which a piece of evidence supports a claim about
cause and effect, within the context of a particular study.
Threats to Internal Validity

There are several threats to internal validity that may exist in an

experiment.
Subject Characteristics
(Selection Bias/Differential Selection) -- The
groups may have been different from the
start. If you were testing instructional
strategies to improve reading and one group
enjoyed reading more than the other group,
they may improve more in their reading
because they enjoy it, rather than the
instructional strategy you used.
Threats to Internal Validity

There are several threats to internal validity that may exist in an

experiment.

Control Group Demoralization

The control group may become
discouraged because it is not receiving
the special attention that is given to the
treatment group. They may perform
lower than usual because of this.
Threats to Internal Validity

There are several threats to internal validity that may exist in an

experiment.
Treatment Diffusion
Sometimes the control group actually
implements the treatment. If two different
techniques are being tested in two different
third grades in the same building, the
teachers may share what they are doing.
Unconsciously, the control may use of the
techniques she or he learned from the
treatment teacher.
Threats to Internal Validity
There are several threats to internal validity that may exist in an
experiment.
Hawthorne Effect
The subjects may respond differently just because they
are being studied. The name comes from a classic study
in which researchers were studying the effect of
lighting on worker productivity. As the intensity of the
factory lights increased, so did the worker productivity.
One researcher suggested that they reverse the
treatment and lower the lights. The productivity of the
workers continued to increase. It appears that being
observed by the researchers was increasing
productivity, not the intensity of the lights.
Threats to Internal Validity
There are several threats to internal validity that may exist in an
experiment.
John Henry Effect

One group may view that it is in competition with

the other group and may work harder than they
would under normal circumstances. This
generally is applied to the control group "taking
on" the treatment group. The terms refers to the
classic story of John Henry laying railroad track.
Threats to Internal Validity
There are several threats to internal validity that may exist in an
experiment.

Maturation

There may be natural changes in the subjects

that can account for the changes found in a
study. A critical thinking unit may appear more
effective if it taught during a time when children
are developing abstract reasoning.
Threats to Internal Validity
There are several threats to internal validity that may exist in an
experiment.

Loss of subjects

All of the high or low scoring subject may have

dropped out or were missing from one of the
groups. If we collected posttest data on a day
when the honor society was on field trip at the
treatment school, the mean for the treatment
group would probably be much lower than it
really should have been.
EXPERIMENTAL RESEARCH

• Experimental research allows the identification of causal relationships between

entities or events. Successful experimental research depends on well-defined
research hypotheses that specify the dependent variables to be observed and
the independent variables to be controlled.

• A variety of laboratory and nonlaboratory research methods are available for

human-computer interaction (HCI) researchers or practitioners when studying
interfaces or applications. The most frequently used include observations, field
studies, surveys, usability studies, interviews, focus groups, and controlled
experiments (Shneiderman et al., 2017).
Types of Experimental Research Design
The types of experimental research design are determined by the way the researcher
assigns subjects to different conditions and groups. They are of 3 types, namely; pre-
experimental, quasi-experimental, and true experimental research.

A. Pre-Experimental

In pre-experimental research design, either a group

or various dependent groups are observed for the
effect of the application of an independent variable
which is presumed to cause change. It is the
simplest form of experimental research design and
is treated with no control group.
The pre-experimental research design is further divided
into three types:

One-shot Case Study Research Design:

In this type of experimental study, only
one dependent group or variable is
considered. The study is carried out
after some treatment which was
presumed to cause change, making it a
posttest study.
One-group Pretest-posttest Research
Design: A type of pre-experimental design
where a single case is observed at two time
points, one before the treatment and one after
the treatment.
Static-group Comparison: A type of pre-
experimental design where the difference
between two groups, one of which receives
the treatment and the other does not, after
treatment was measured.
Types of Experimental Research Design

B. Quasi-Experimental
A quasi-experiment is an empirical interventional study
used to estimate the casual impact of an intervention on
target population without random assignment.
Some examples of quasi-experimental research design include; the
time series, no equivalent control group design, and the
counterbalanced design.

Nonequivalent Pretest-Posttest Design

With this design both a control group and an

experimental group is compared, however,
the groups are chosen rather than through
randomization.
Time-Series Design

Time series designs refer to the pretesting

and post testing of one group of subjects
at different intervals.
Nonequivalent Before-After Design

This design is used when we want to

compare two groups that are likely to be
different even before the study begins.
Types of Experimental Research Design

C. True Experimental
The true experimental research design relies on
statistical analysis to approve or disprove a hypothesis.
It is the most accurate type of experimental design and
may be carried out with or without a pretest on at least
2 randomly assigned dependent subjects.
The classification of true experimental design include:

Posttest Equivalent

Each group is presented with the treatment

and posttests are then given to each subject
to determine if a difference between the two
groups exists.
The classification of true experimental design include:

Pretest Posttest Equivalent

This design provides for both a control

group and a measure of change but also
adds a pretest to assess any differences
between the groups.
The classification of true experimental design include:

Solomon Group

The Solomon four group test is a

standard pretest-posttest two-group
design and the posttest only control
design.
If you have any question, kindly leave a
comment on the discussion for Module
5.