Pinto, Razmen R.-InAndEx

Two General
Criteria of Research
Designs
Razmen R. Pinto
OVERVIEW
Here’s what you’ll find in my report:

 Internal Validity vs. External Validity
 Trade-off between Internal and External Validity
 Threats to Internal Validity and how to counter them
 Types of External Validity
 Threats to External Validity and how to counter them
INTERNAL VALIDITY vs. EXTERNAL VALIDITY
Internal validity External validity

 refers to the degree of  refers to the extent to
confidence that the causal which results from a
relationship being tested study can be applied
is trustworthy and not (generalized) to other
influenced by other situations, groups or
factors or variables. events.
Trade-off between Internal and
External Validity
There is an inherent trade-off between internal and

external validity; the more you control extraneous factors
in your study, the less you can generalize your findings to
a broader context.
Example:
You want to test the hypothesis that drinking a cup of coffee improves memory.
You schedule an equal number of college-aged participants for morning and
evening sessions at the laboratory. For convenience, you assign all morning
session participants to the treatment group and all evening session
participants to the control group.
Once they arrive at the laboratory, the treatment group participants are given a cup
of coffee to drink, while control group participants are given water. You also
give both groups memory tests. After analyzing the results, you find that the
treatment group performed better than the control group on the memory test.
How to check whether your
study has internal validity?
Example:
You want to test the hypothesis that drinking a cup of coffee improves memory.
You schedule an equal number of college-aged participants for morning and
evening sessions at the laboratory. For convenience, you assign all morning
session participants to the treatment group and all evening session
participants to the control group.
Once they arrive at the laboratory, the treatment group participants are given a cup
of coffee to drink, while control group participants are given water. You also
give both groups memory tests. After analyzing the results, you find that the
treatment group performed better than the control group on the memory test.
Conditions
1 2 3
Your treatment and response Your treatment precedes No confounding or extraneous

variables change together. changes in your response factors can explain the results
variables of your study.
 Drinking coffee and  Drinking coffee × The time of day of the

memory performance happened before the sessions is an extraneous
increased together. memory test. factor that can equally
explain the results of the
study.
Threats to Internal
Validity and How to
counter them
The management of company X wants to know if flexible
working hours will improve job satisfaction among
employees. They set up an experiment with two groups: 1)
control group of employees with fixed working hours 2)
experiment group with employees with flexible working
hours. The experiment will run for six months. All employees
fill in a survey measuring their job satisfaction before the
experiment (pre-test) and after the experiment (post-test).
—Research example
Threats
History Maturation Testing

An unrelated event The outcomes of the study The pre-test (used to
influences the outcomes. vary as a natural result of establish a baseline)
time. affects the results of the
post-test.
Participant selection
Participants in the control and experimental
group differ substantially and can thus not
be compared.
Threats
Attrition Regression towards Instrumentation

Over the course of a (longer) mean There is a change in how the
study, participants may drop Extreme scores tend to be dependent variable is
out. If the drop out is caused closer to the average on a measured during the study.
by the experimental treatment second measurement.
(as opposed to coincidence) it
can threaten the internal
validity.
Social interaction
Interaction between
participants from different
groups influences the
outcome.
How to
counter
threats to
Internal
Validity
Threats
Maturation
History If feasible within your Testing
Evaluators should do their evaluation questions, reducing Keep an eye out for this threat
best to identify any external the amount of time between whenever there is a pretest-
events or changes that may the pretest and posttest can posttest design and no
impact their program results limit maturation threats comparison group to help
control for the learning curve
of taking the pretest.
Participant selection
Be alert for this potential threat if you are
working with a nonequivalent comparison
group.
Threats
Attrition Regression towards Instrumentation

Evaluators should seek to mean Keep an eye out for this if
employ intensive strategies to Avoid selecting participants there are multiple
minimise study attrition such based on extreme performance observation/test points in your
as through offering incentives, or scores study
alternative locations for data
collection and repeat
reminders via mail
Social interaction
The threats described here can
often be minimized by
constructing multiple groups
that are not aware of each
other
Types of External
Validity
There are two main types of external

validity: population validity and
ecological validity.
1. Population validity
Example
You want to test the hypothesis that people tend to

perceive themselves as smarter than others in terms
Population validity refers to whether you of academic abilities. Your target population is the
can reasonably generalize the findings 10,000 undergraduate students at your university.
from your sample to a larger group of
people (the population). You recruit over 200 participants. They are science
and engineering majors; most of them are Filipino,
male, 18–20 years old and from a high
socioeconomic background. In a laboratory setting,
you administer a mathematics and science test and
then ask them to rate how well they think performed.
You find that the average participant believes they
are smarter than 66% of their peers.
Example
You want to test the hypothesis that driving
reaction times become slower when people pay
attention to others talking.
2. Ecological validity In a laboratory setting, you set up a simple
computer-based task to measure reaction times.
Participants are told to imagine themselves
Ecological validity refers to whether driving around the racetrack and double-click
you can reasonably generalize the the mouse whenever they see an orange cat on
findings of a study to other situations the screen. For one round, participants listen to a
and settings in the ‘real world’. podcast. In the other round, they do not need to
listen to anything. After assessing the results,
you find that reaction times are much slower
when listening to the podcast.
Threats to
External Validity
and How to
counter them
Example
A researcher wants to test the hypothesis that
people with clinical diagnoses of mental
disorders can benefit from practicing mindfulness
daily in just two months time. They recruit people
who have been diagnosed with depression for at
least a year, are aged between 20–29, and live
locally.
Participants are given a pretest and a post-test
measuring how often they experienced anxiety in
the past week. During the study, all participants
are given an individual mindfulness training and
asked to practice mindfulness daily for 15
minutes in the morning.
Since the levels of anxiety decreased between

the pre- and post-test, the researcher concludes
that all clinical populations can benefit from
mindfulness.
Threats to External Validity
Experimenter
Sampling Bias History
effect
The characteristics or
The sample is not behaviors of the
An unrelated event
representative of the experimenter(s)
influences the outcomes.
population. unintentionally influence
the outcomes.
Threats to External Validity
Hawthorne Aptitude- Situation

Testing effect
effect treatment effect
The tendency for

participants to Interactions between Factors like the
The administration characteristics of the setting, time of day,
change their
of a pre- or post- group and individual location, researchers’
behaviors simply variables together characteristics, etc.
test affects the
because they know influence the limit generalizability
outcomes.
they are being dependent variable. of the findings.
studied.
How to counter
Threats to
External Validity
How to counter threats to external validity
Field Probability
Replications Recalibration
Experiments Sampling
Probability sampling
Replications counter Recalibration or
Field experiments counters selection
almost all threats by reprocessing also
counter testing and bias by making sure
enhancing counters selection
everyone in a
generalizability to situation effects by population has an
bias using algorithms
other settings, using natural to correct weighting
equal chance of being
populations and contexts. of factors (e.g., age)
selected for a study
conditions. within study samples.
sample.

Pinto, Razmen R.-InAndEx

Uploaded by

Copyright:

Available Formats

Pinto, Razmen R.-InAndEx

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pinto, Razmen R.-InAndEx

Uploaded by

Copyright:

Available Formats

Two General

Here’s what you’ll find in my report:

Internal validity External validity

There is an inherent trade-off between internal and

Your treatment and response Your treatment precedes No confounding or extraneous

 Drinking coffee and  Drinking coffee × The time of day of the

History Maturation Testing

Attrition Regression towards Instrumentation

Attrition Regression towards Instrumentation

There are two main types of external

You want to test the hypothesis that people tend to

Since the levels of anxiety decreased between

Hawthorne Aptitude- Situation

The tendency for

You might also like