Topic 5

Download as pptx, pdf, or txt
Download as pptx, pdf, or txt
You are on page 1of 29

ICT 615

Information Technology Research


Methods

Topic 5
Experimental design
Acknowledgement
of Country
We acknowledge the Binjareb people of the Noongar nation as
the traditional custodians of this country and its waters and that
Murdoch University stands on Noongar country.

We pay our respects to Noongar elders, past and present and


acknowledge their wisdom and advice in teaching and cultural
knowledge activities.
COVID-19
student protocols.
Personal If you’re unwell don’t come
Visit MyMurdoch Student Portal to stay up-to-date. to campus. Stay home, get
Health tested and see a doctor.

If you’re diagnosed with


Please wipe down all
Computer COVID-19 or have been in close

Labs
surfaces after you have Reporting contact with a confirmed case,
used them. stay home and let your unit
coordinator know.

Be kind, respectful and


Behaviour supportive to people in Physical Stay 1.5 metres apart
at all other times.
our University community. Distancing

Leave space for people Wash your hands with


Entering soap and water regularly,
+ Exiting
when entering and Hygiene and cough and sneeze
leaving buildings.
into your elbow.
Readings and resources

Required reading:
• Oates Ch 9

Recommended reading:
• Basili, V. R. & Zelkowitz, M. V. (2007) Empirical studies to build a science of
computer science. Communications of the ACM 50(11) 33-37.

4
Learning objectives

After completing this topic you should be able to:


• Explain what an experiment is
• Define the following terms in the context of ICT experiments: treatment, control
group, randomisation
• Explain what an experimental design is
• Describe the advantages and disadvantages of experiment

5
Experiments are a research strategy or approach

6
What is an experiment?

An experiment involves observation under controlled


conditions to test a hypothesis.
An experiment is a research strategy that investigates
cause and effect relationships (causality).
All factors that might affect the results, other than the
factors that are hypothesised to cause a particular
outcome, are excluded/controlled.

7
Treatment

• In an experiment, the factor (also called an independent variable) is an


explanatory variable manipulated by the experimenter. Each factor has two or
more levels, i.e., different values of the factor. Combinations of factor levels are
called treatments.

https://stattrek.com/statistics/dictionary.aspx?definition=treatment

8
REVISION - Hypotheses

A hypothesis is a statement about the relationships between variables. For


example:
H1: Pair programming improves program quality
H2: Users who received password memorisation training will create stronger
passwords

Experiments are designed to try to prove or disprove hypotheses

9
Causality versus correlation

Correlation is where there is a relationship between 2


variables.
Causality is when the change in one variable is the cause
of a change in another variable.

See the following animation for a good explanation of the


difference:
http://www.abs.gov.au/websitedbs/a3121120.nsf/home/stat
istical+language+-+correlation+and+causation

10
Question

Which of the following hypotheses express a causal relationship?


H1: Level of use is associated with user satisfaction with a system
H2: Pair programming improves program quality
H3: Users who receive password memorisation training will create stronger
passwords

11
Characteristics of experiments

• Precise observations and measurements


• Pre-test and post-test observations
• Proving or disproving hypotheses
• Identification of causal factors, i.e. one-directional links
• Explanation and prediction
• Repetition

Cause Effect

Independent Dependent
variable variable
12
Control variables and the environment

Ways of controlling factors that might affect the outcomes of an experiment include:
• Eliminate the factor via exclusion criteria
• Hold the factor constant via inclusion criteria (if you cannot eliminate it)
• Use random selection/allocation
• Use control group - the only difference from treatment group is the change in independent variable.

Unwanted
factor

For example: Effect


Cause 13
Validity

Validity is concerned with the study's success at measuring what the researchers
set out to measure. Researchers should be concerned with both external and
internal validity.
Internal validity relates to the rigour with which the study was conducted. An
experiment has good internal validity if the results are due to the manipulations of
the independent variable, and not to any other factors.
External validity refers to the extent to which the results of a study are
generalizable or transferable – to different times, places, technologies etc.

14
Internal validity threats

Threats to internal validity include:


• Differences between experimental and control group
• History, i.e. "what has happened in between"
• Maturation, due to age, practice, boredom etc.
• Instrumentation, e.g. fault or change in measurement instrument
• Experimental mortality, i.e. changes in group composition over time
• Reactivity and experimenter effects, e.g. "behaving correctly".

15
External validity threats

Threats to external validity include:


• Using samples that are not representative of the population – e.g.
students
• Too few participants

16
Types of experiment

Experimental design refers to the way in which the


experiment is designed – e.g. use of randomisation or not,
use of control groups or not, number of experimental
groups, pretesting of participants etc.
Different experimental designs vary in their power to
control contamination of the relationship between
independent and dependent variables.
• “True" (lab) experiments - high control but possibly
unrealistic settings
• Quasi-experiments or field experiments – more
realistic settings but less control

17
Example experimental designs

Pre-experimental designs fail to control threats to internal


validity - e.g.
X 0
No control; no pretest
0 X 0
Pretest and post-test BUT no control => can't control for
history, maturation etc.
X 0
--------
0
Control (no randomization) BUT no pretest (=> can't tell if
groups are really comparable)

18
Example experimental designs

True experimental design - e.g. post-test only control


group design
R 01 X 02
R 03 04
where effect of experimental variable is E= (02 - 01) - (04 -
03)

R X 0
R 0
Assumes that randomisation removes the need for
pretesting, and because subjects are only tested once
reduces threats due to testing and instrumentation

19
Example experimental designs

Quasi-experiment design e.g. non-equivalent control


group design
0 X 0
--------
0 0

No randomization. Groups may be naturally assembled


(e.g. classes) or self-selected (volunteers used to form
experimental group)

20
Causality

The following conditions must be met to establish causality:


• Correlation - relationship between independent and dependent variable
• Time order - cause precedes effect in time (values of IV change then values of
DV change)
• Rule out other possible explanations

21
Designing and conducting an experiment

1.Select relevant variables


2.Specify levels of treatment
3.Ensure control of experimental environment
4.Choose the experimental design
5.Select and assign subjects
6.Pilot test, revise, test - pretesting of procedures as well
as instruments
7.Analyse data

22
Field Experiment

• Can be used for both theory testing and theory building. Field experimentation is
of value because it enables the development of causal models based on
information gathered in natural, and hence more generalizable settings, settings
for studying phenomena that could not easily be replicated in a laboratory setting.

• Ranges from true experimental designs to quasi experimental designs.

23
Problems with conducting field experiments

• 1. Managers, participants or researchers resist random assignment - may need to


eliminate no treatment control group such that all participants receive a treatment
- means can't determine absolute impact of the treatment but can draw
conclusions about the differential effects of the various treatments

• 2. Unintended treatment occurs in the control group - eg. control group


overcompensation, or control group demoralisation - requires careful monitoring

24
Problems with conducting field experiments

3. Participants withdraw because of treatment received - conduct exit interviews,


compare background on those who drop out with those who stay to determine if
they are different

4. Treatment implementation variance

5. Experimental group contamination - due to contact between groups. Attempt to


monitor contact between groups.

25
Advantages
Some advantages of experiments as a research strategy
include:
• Well-established method
• The only way to demonstrate cause-effect relationships
• Ability to repeat experiments and extend results across people, times &
situations

26
Disadvantages
Some disadvantages include:
• Artificiality – can limit ability to generalise
• Often impossible to control all the parameters
• Difficult to recruit representative samples
• Ethical considerations – control group not receiving treatment believed to
be valuable

27
Learning objectives revisited

• What is an experiment?
• Why would a control group be used in an experiment?
• What is the difference between a true experiment and a
quasi-experiment?
• What is an experimental design?
• What is the main advantage of experimentation as a
research strategy?

28
Thank you

You might also like