Experimental Designs

Y520
Strategies for Educational Inquiry

Experimental designs-1

Research Methodology

■ Is concerned with how the design is implemented and how the

research is carried out. The methodology used often determines
the quality of the data set generated. Methodology specifies:
• When and how often to collect data
• Construction of data collection measures
• Identification of the sample or test population
• Choice of strategy for contacting subjects
• Selection of statistical tools
• Presentation of findings

Experimental designs-2

1
Categories of Research Design

■ Pre – experimental (descriptive) designs

■ Quasi – experimental designs
■ Correlational & Ex-Post-Facto designs
■ Causal-comparative designs
■ (True) Experimental designs

Experimental designs-3

Pre-experimental Designs

■ One-shot case studies

■ One group, pre-test, post-test design
■ Static group comparison

Experimental designs-4

2
Quasi-experimental Designs

■ Non-randomized, control group, pre-test,

post-test
■ Time series
• Time series with control group
• Equivalent time series
■ Pre-test, post-test control group

Experimental designs-5

Correlational & Ex Post Facto Designs

■ “Ex-Post-Facto” are also called “causal-

comparative” designs.
■ Correlational designs
■ Ex-Post-Facto designs

Experimental designs-6

3
(True) Experimental Designs

■ Randomized post-test only, control group.

■ Randomized pre- and post-test, control group.
■ Randomized Solomon Four Group
■ Randomized assignment with matching
■ Randomized pre- and post-test, control group,
matching.

Experimental designs-7

Why use experimental designs?

Man prefers to believe what he prefers

to be true.

— Francis Bacon

Experimental designs-8

4
Why use experimental designs?

The best method — indeed the only fully

compelling method — of establishing causation
is to conduct a carefully designed experiment in
which the effects of possible lurking variables
are controlled. To experiment means to actively
change x and to observe the response in y.

— Moore, D., & McCabe, D. (1993). Introduction to the practice of

statistics. New York: Freeman. (p.202).

Experimental designs-9

Why use experimental designs?

The experimental method is the only method of

research that can truly test hypotheses
concerning cause-and-effect relationships. It
represents the most valid approach to the
solution of educational problems, both practical
and theoretical, and to the advancement of
education as a science.
— Gay, L. R. (1992). Educational research (4th Ed.). New York:
Merrill. (p.298).

Experimental designs-10

5
Why use experimental designs?

To propose that poor design can be corrected

by subtle [statistical] analysis techniques is
contrary to good scientific thinking.

— Pocock, Stuart (19xx). Controlled clinical trials. (p.58).

Experimental designs-11

Why use experimental designs?

Issues of design always trump issues of

analysis.

— G. E. Dallal, 1999, explaining why it would be wasted effort to

focus on the analysis of data from a study whose design was
fatally flawed. (http://www.tufts/edu/~gdallal/study.htm)

Experimental designs-12

6
Why use experimental designs?

100% of all disasters are failures of design, not

analysis.

— Ron Marks, Toronto, August 16, 1994.

Experimental designs-13

Unique Features of Experiments

■ Investigator manipulates a variable directly

(the independent variable), while controlling
all other potentially influential variables.

■ Empirical observations from experiments

provide the strongest basis for inferring
causal relationships.

Experimental designs-14

7
Additional Features of Experiments (a)

■ ¿ ¿
Problem statement theory constructs ¿
¿
operational definitions variables ¿
hypotheses.
■ The research question (hypothesis) is often
stated as an alternative to the null hypothesis.
■ Random sampling of subjects (insures
sample is representative of population)

Experimental designs-15

Additional Features of Experiments (b)

■ Random assignment of subjects to treatment

and comparison groups (insures equivalency
of groups; i.e., unknown variables that may
influence outcome are distributed randomly
across groups).

■ After treatment, performance (dependent

variable) of subjects in both groups is
compared.

Experimental designs-16

8
Additional Features of Experiments (c)

■ Whenever an experimental design is used,

the researcher is interested in determining
cause and effect.

■ The causal variable is the independent

variable and the outcome or effect is the
dependent variable.

Experimental designs-17

Independent Variable Manipulation (a)

■ Presence or Absence technique.

• An independent variable can be
manipulated by presenting a condition or
treatment to one group (treatment or
experimental) and withholding the
condition or treatment from another group
(comparison or control).

Experimental designs-18

9
Independent Variable Manipulation (b)

■ Amount technique.
• The independent variable can be
manipulated by varying the amount of a
condition, such as varying the amount of
practice a child is permitted in learning a
complex skill (such as learning to play the
piano).

Experimental designs-19

Independent Variable Manipulation (c)

■ Type of Treatment technique.

• The independent variable can be
manipulated by varying the type of a
condition or treatment administered. For
example, students in one group outline
reading material. Students in another
group write a summary sentence for each
paragraph. Students in a third group draw
concept maps.

Experimental designs-20

10
Controlling Extraneous Variables (a)

■ The goal: All groups (treatment and comparison) are

equivalent to each other on all characteristics or
variables at the beginning of an experiment.
■ The only systematic difference between the groups in an
experiment is the presentation of the independent
variable. The groups should be the equivalent on all
other variables.
■ Suppose an investigator wanted to test the effect of a
new method of spatial training. If one group is mostly
females and the other group is mostly males, then the
variable of gender may differentially affect the outcome.

Experimental designs-21

Controlling Extraneous Variables (b)

■ Confounding variables (such as gender) are controlled

by using one or more procedures that eliminate the
differential influence of an extraneous variable.
■ Random assignment of subjects to groups. This is
the best means of controlling extraneous variables in
experimental research. Extraneous variables are
assumed to be distributed randomly across groups.
■ Random assignment controls for both known and
unknown confounding variables.

Experimental designs-22

11
Controlling Extraneous Variables (c)

■ Random assignment results in groups that are

equivalent on all variables at the start of the
experiment.
■ Subjects should be randomly selected from from
a population and then randomly assigned to
groups.
■ Random selection ensures external validity.
■ Random assignment ensures internal validity.

Experimental designs-23

Controlling Extraneous Variables (d)

■ Random assignment is more important than random

selection because the goal of an experiment is to
establish firm evidence of a cause and effect
relationship.
■ Random assignment controls for the differential
influence that confounding extraneous variables may
have in a study by insuring that each participant has an
equal chance of being assigned to each group.
■ In addition, the equal probability of assignment means
the characteristics of participants are also equally likely
to be assigned to each group.
■ Participants and their characteristics should be
distributed approximately equally in all groups.
Experimental designs-24

12
Controlling Extraneous Variables (e)

■ Extraneous variables may exist even after

random sampling and random assignment
(i.e., the groups are not equivalent):
• Subject mortality (dropouts due to
treatment)
• Hawthorne effect
• Fidelity of treatment (manipulation check)
• Data collector bias (double blind studies)
• Location, history
Experimental designs-25

Controlling Extraneous Variables (f)

■ Additional procedures for controlling

extraneous variables (used as needed):
• Use subjects as own control
• Matching subjects on certain
characteristics
• Blocking
• Analysis of covariance

Experimental designs-26

13
Controlling Extraneous Variables (g)

■ Matching
• Match participants in the various groups on the
extraneous variable that needs to be controlled.
• This eliminates any differential influence of that
variable.
• Procedure: Identify the confounding extraneous
variables to be controlled, measure all participants
on this variable, find another person who has the
same amount or type of these variables, and then
assign these two individuals, one to the treatment
group and one to the comparison group.

Experimental designs-27

Controlling Extraneous Variables (h)

■ Hold the extraneous variable constant

• Controls for confounding extraneous
variables by insuring that the participants
in the different groups have the same
amount or type of a variable.
• For example, in a study of academic
achievement, participants are limited to
people who have an IQ greater than 120.

Experimental designs-28

14
Controlling Extraneous Variables (i)

■ Blocking
• Controls for a confounding extraneous variable by
making it an independent variable.
• For example, in a study of academic achievement,
participants in one (comparison and treatment)
group consists only of people who have an IQ
greater than 120. In another block of comparison
and treatment groups, are participants who have
IQ’s between 80 and 120. And so on.

Experimental designs-29

Controlling Extraneous Variables (i)

■ Counterbalancing
• Used to control for order and carry-over effects. Is
relevant only for a design in which participants
receive more than one treatment (e.g., repeated
measures).
• Order effects: Sequencing effects that arise from
the order in which people take the various
treatment conditions.
• Carry-over effects: Sequencing effects that occur
when the effect of one treatment condition carries
over to a second treatment condition.

Experimental designs-30

15
Controlling Extraneous Variables (j)

■ Counterbalancing
• Controls for order and carry-over effects by
administering each experimental treatment
condition to all groups of participants but in
different orders.

Experimental designs-31

Controlling Extraneous Variables (k)

■ Analysis of Covariance
• This is a statistical technique used when
participants in various groups differ on some
variable that could affect their performance.
• Example, If students with higher GRE Verbal
scores were in one of two groups, these students
would be expected to learn faster. Thus you would
want to control for verbal skill (assuming that is
what GRE Verbal represents).

Experimental designs-32

16
Controlling Extraneous Variables (l)

■ Analysis of Covariance
• Analysis of covariance statistically adjusts
the dependent variable scores for the
differences that exist in an extraneous
variable.
• The only relevant extraneous variables are
those that also effect participants’
responses to the dependent variable.

Experimental designs-33

Experimental Designs

■ Pre-Test / Post-test comparison group

■ Post-test only comparison group

■ Solomon Four-Group

Experimental designs-34

17
 Randomized,
Pre-test / Post-test, Control Group Design

Treatment R O1 X1 O2
Comparison R O1 O2

R = Random assignment
X = Treatment occurs for X1 only
O1= Observation (Pre-test)
O2= Observation (Post-test, dependent variable)
(Random sampling assumed)
Experimental designs-35

Pre-test / Post-test control group design

■ Subjects (aka, participants) are randomly assigned to

the experimental and comparison groups.
■ Both groups are pre-tested on the dependent
variable, the treatment (aka, independent variable) is
administered to the experimental group, and both
groups are then post-tested on the dependent
variable.

Experimental designs-36

18
Pre-test / Post-test control group design

■ This is a strong design because

• subjects are randomly assigned to groups,
insuring equivalence;
• a comparison group is used;
• most of the potentially confounding variables that
might threaten internal validity are controlled.

Experimental designs-37

Pre-test / Post-test control group design

Controls:
■ Random sampling from population.
■ Random assignment of Ss to experimental &
comparison groups.
■ Timing of independent variable.
■ All other variables / conditions that might influence
the outcome are controlled.

Experimental designs-38

19
Control: Random Sampling from Population

■ Guarantees all Ss equally likely to selected.

■ May assume the sample is representative of the

population on all important dimensions.

■ No systematic differences between sample and

population.

Experimental designs-39

Control: Random Assignment to Groups

■ Occurs after random sampling of subjects.

■ Guarantees all Ss equally likely to be in comparison or
experimental group.
■ May assume the two groups are equivalent on all
important dimensions.
■ No systematic differences between groups.
■ May substitute matching on characteristics that might
affect dependent variable (e.g., IQ, income, age).
■ May not know which characteristics to match on –
compromises internal validity.

Experimental designs-40

20
Control: Independent Variable

■ Both groups experience the same conditions, except

the experimental group is exposed to independent
variable, in addition to the shared conditions.

■ Experimenter controls when and how the independent

variable is applied to experimental group.

Experimental designs-41

Summary:
Steps in Pre-Test / Post-test Controlled Exp.

■ Random sampling of subjects.

■ Random assignment of subjects to comparison or
experimental groups.
■ Administer pre-test to all subjects in both groups.
■ Ensure both groups have same conditions, except that
experimental group receives the treatment.
■ Administer post-test to all subjects in both groups.
■ Assess change in dependent variable from pre-test to
post-test for each group.

Experimental designs-42

21
 Randomized,
Pretest / Post-test,Control Group Design

Treatment R O1 X1 O2
Comparison R O1 O2

R = Random assignment
X = Treatment occurs for X1 only
O1= Observation (Pre-test)
O2= Observation (Post-test, dependent variable)

Experimental designs-43

Randomized
Pretest / Post-test, Control Group Design

Random O1= (Pre-test) Exposure to O2= (Post-test)

Assignment of Of dependent Treatment (X) Of dependent
Ss to: variable independ var variable

Exp group’s Exp group’s

Experimental average score X average score
Group on dep var on dep var

Comparison Comparison
Comparison group’s group’s
Group average score average score
on dep var on dep var
Experimental designs-44

22
 Randomized Pretest / Post-test:
Within Group Differences (a)

Control group Control group Control group difference

— =
Pre-test Score Post-test Score on dependent variable

Experimental gp Experimental Experimental group

— gp Post-test = difference on dependent
Pre-test Score Score variable

Experimental designs-45

Randomized Pretest / Post-test:

Within Group Differences (b)

The difference in the control group’s score from the pre- to post-
test shows the amount of change in the dependent variable that is
expected to occur without exposure to the independent or
treatment variable.

The difference in the experimental group’s score from pre- to post-

test shows the amount of change in the dependent variable that
could be expected to occur with exposure to the independent or
treatment variable.

Experimental designs-46

23
 Randomized Pre-test / Post-test:
Between Group Differences

Control group Experimental Difference attributable to

— =
difference group difference independent variable

The difference between the change in the experimental group and

the change in the control group is the amount of change in the
dependent variable that can be attributed to the influence of the
independent or treatment variable.

Experimental designs-47

Example:
Pre-Test / Post-test Control Group Design

■ An educator wants to compare the effectiveness of

computer software that teaches reading with that of a
standard reading curriculum.
■ She tests the reading skill of a group of 60 fifth
graders, and then randomly assigns them to two
classes of 30 students each.
■ One class uses the computer reading program while
the other studies a standard curriculum.
■ In June she retests the students and compares the
average increase in reading skill in each of the two
classes.

Experimental designs-48

24
 Example:
Pre-test / Post-test Control Group Design

Random Exposure to
Score on test Score on test
Assignment of Treatment (X)
of reading skill of reading skill
Ss to: independ var

Computer Average X Average

(Exp Group) Score: 68 Score: 91

Comparison Average Average

Group score:89 score: 104

Experimental designs-49

Example:
Pre-Test / Post-test Control Group Design

■ The comparison group’s average reading score was

89 on the pre-test and 104 on the post-test. The
difference between pre- and post-tests is 15 points.
This is the amount of change in reading skill that could
be expected to occur with the regular curriculum only.
■ The experimental group’s average reading score was
68 on the pre-test and 91 on the post-test. The
difference between pre- and post-tests is 23 points.
The change for the experimental group from pre- to
post-test indicates the the change in reading skill that
could be expected to occur with the computer reading
curriculum.

Experimental designs-50

25
Example:
Pre-Test / Post-test Control Group Design

■ The difference between the change in the

experimental group (23 points) and the change in the
comparison group (15 points) is 8 points. This is the
amount of change in reading skill that can be attributed
solely to the computer reading program.

Experimental designs-51

Example: Rosenthal (1966)

Pre-Test / Post-test Control Group Design

■ Recall this experiment

Experimental designs-52

26
Post-test only control group design

■ This design includes all the same steps as the pre-

test/post-test design except that the pre-test is
omitted.
■ Reasons for omitting pre-testing:
• Subjects already exposed to the treatment
• Too expensive or too time-consuming
■ With large groups, this design controls most of the
same threats to internal & external validity. For small
groups, a pre-test is necessary.
■ Eliminates the threat of pre-testing
■ May decrease experimental mortality by shortening
the length of the study. Experimental designs-53

Randomized Post-test, Control Group Design

Treatment R X1 O
Comparison R O

R = Random assignment
X = Treatment occurs for X1 only
O = Observation (dependent variable)
(Random sampling assumed)

Experimental designs-54

27
Example:
Randomized Post-test, Control Group

■ Chase 1976

Experimental designs-55

Randomized Solomon Four Group Design

Treatment R O1 X1 O2
Comparison R O1 X2 O2

Treatment R X1 O2
Comparison R X2 O2

R = Random assignment
X = Treatment occurs for X1 only
O1= Observation (Pre-test)
O2= Observation (Post-test, dependent variable)

Experimental designs-56

28
 Randomized Assignment with Matching

Treatment M,R X1 O

Comparison M,R O

M = Matched Subjects
R = Random assignment of matched pairs
X = Treatment (for X1 only)
O = Observation (dependent variable)

Experimental designs-57

Example:
Randomized Assignment with Matching

■ An experimenter wants to test the impact of a novel instructional

program in formal logic. He infers from previous research that
high ability students and those with programming, mathematical,
or musical backgrounds are likely to excel in formal logic
regardless of type of instruction.
■ Experimenter randomly samples subjects, looks at subjects SAT
scores, matches subjects on basis of SAT scores, and randomly
assigns matched pairs (one of each pair to each group).
■ Other concominant variables (previous programming,
mathematical, and musical experience) could also be matched.
■ Difficult to match on more than a few variables.

Experimental designs-58

29
 Randomized Pretest – Post-test
Control Group, Matched Ss

Treatment O1 M,R X1 O2

Comparison O1 M,R O2

O1= Pretest
M = Matched Subjects
R = Random assignment of matched pairs
X = Treatment (for X1 only)
O2= Observation (dependent variable)

Experimental designs-59

Methods of Matching

■ Mechanical

■ Statistical

Experimental designs-60

30
Mechanical Matching

■ Rank order subjects on variable(s), take top

two, randomly assign members of pair to
groups. Repeat for all pairs.
■ Problems:
• Impossible to match on more than one or
two variables simultaneously.
• May need to eliminate some Ss because
there is no appropriate match for one of
the groups.
Experimental designs-61

Statistical Matching

■ Used to control for factors that cannot be

randomized by nonetheless can be measured
on an interval scale.
■ Achieved by measuring one or more
concomitant variable (the “covariate”) in
addition to the variable (“variate”) of interest
(the dependent variable).
■ Can be used in experimental, quasi-
experimental, and non-experimental designs.

Experimental designs-62

31
Factorial Designs (a)

■ To compare the effects of two or more independent

variables in the same experiment, cross them.
■ Factorial designs enable the researcher to determine
the independent influence of each independent
variable, and the interactive influence of the
independent variables.
■ Each combination of independent variables is called
a “cell.”
■ Subjects are randomly assigned to as many groups
as there are cells.

Experimental designs-63

Factorial Designs (b)

■ Groups of subjects are administered the

combination of independent variables that
corresponds to their assigned cell; then their
performance is measured.
■ Data collected from factorial designs yields
information on the effect of each independent
variable separately and the interaction
between (and/or among) the independent
variables.

Experimental designs-64

32
Factorial Designs (c)

■ Main effect: The effect of each independent

variable on the dependent variable.
• Each independent variable may have a
main effect.
• If a study included the independent
variables “type of instruction” and “student
verbal skill,” then, potentially, there are two
main effects, one for instruction type, and
one for verbal skill.

Experimental designs-65

Factorial Designs (d)

■ Interaction:
• Occurs between two or more independent
variables when the effect that one independent
variable has on the dependent variable depends
on the level of the other independent variable.
• If gender is an independent variable and method
of teaching mathematics is another independent
variable, then an interaction exists if:
– the lecture method was more effective for teaching males
and
– individualized instruction was more effecting in teaching
females mathematics.

Experimental designs-66

33
Example:
2 x 2 Factorial Design (f)

Cells = Conditions = Groups

Reinforcement

Immediate Delayed
Low
Ses
Middle

Experimental designs-67

Example:
2 x 2 Factorial Design (g)

Results
Reinforcement

Immediate Delayed
Low M11 = 40 M21 = 15
Ses
Middle M12 = 40 M22 = 40

Experimental designs-68

34
Example:
2 x 2 Factorial Design (h)

Interaction
50
Immediate
40

Mean 30
Vocabulary
Delayed
Score 20

10

0
Low Middle
SES

Experimental designs-69

Repeated Measures Design

■ In the simplest repeated measures design, each

participant receives all treatment conditions but they
receive different sequences of treatments (usually
randomly determined).
■ Each participant is compared with him or herself.
Even subtle treatment effects may be statistically
significant. Hence, the main advantage of this design
is power – the ability to detect small differences.
■ The main problem is detecting and dealing with order
effects (practice effects, fatigue effects, carryover
effects, and sensitization).
■ Counterbalancing: used to balance out routine order
effects.
Experimental designs-70

35
Repeated Measures Design

■ Suppose a researcher were investigating the

effect of type of instruction on learning
mathematics.
■ Lecture and individualized instruction were
the two types.
■ All participants would receive both types of
instruction but some participants would
receive the lecture first and then
individualized instruction, while other
participants would receive the reverse order.

Experimental designs-71

Counterbalancing:
Repeated Measures Design

Type of Instruction
Group A Lecture Lecture
Test Test Test
Group B Individual Inst Individual Inst

Experimental designs-72

36
Counterbalancing:
Repeated Measures Design

■ This design permits us to answer:

• Is the lecture method more effective than
the individual instruction? (Main effect of within-subjects
factor of type of instruction)

• Do students learn more if they experience

the lecture first and then the individualized
instruction, or the inverse? (Main effect of the between
subjects factor of counterbalancing sequence)

• Is performance better after the second type

of instruction than the first? (Instruction by
counterbalancing interaction)

Experimental designs-73

37