Research Methods, Design,

and Analysis
Eleventh Edition

CHAPTER 6
Research Validity

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Research Validity

• Research validity refers to the correctness

or truthfulness of an inference that is made
from the results of a research study.
– Four Major Types of Validity
 Statistical conclusion validity
 Construct validity
 Internal validity
 External validity

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
Statistical Conclusion Validity

• Do independent and dependent variables

covary?
• Inferential statistics allow us to establish
this type of validity
• Small sample size is a threat to statistical
conclusion validity

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Construct Validity

• Definition—extent to which we can infer

higher-order constructs for our operations
• Constructs are used for
– research participants
– independent variable
– dependent variable
– experimental setting

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Construct Validity

• Inadequate explanation of the construct

• Construct confounding
• Mono-operation bias
• Confounding constructs with level of
constructs
• Treatment-sensitive factorial structure
• Reactive self-report changes
• Reactivity to the experimental situation
Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Construct Validity
(cont'd)
• Experimenter effects
• Novelty and disruption effects
• Compensatory equalization
• Compensatory rivalry
• Treatment diffusion

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Construct Validity
(cont'd)
• Reactivity to the Experimental Situation
• Refers to research participants’ motives
and perceptions influencing their response
to the DV
– motive and perceptions influenced by the
demand characteristics of the experiment
– primary motive--positive self-presentation
– condition producing positive self-presentation
motive
Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Construct Validity
(cont'd)
• Implication for research
– intertreatment interaction
– intratreatment interaction
• Experimenter Effect
• Experimenter’s motive of supporting the
study hypothesis can lead to bias

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Construct Validity
(cont'd)
• Ways experimenter may bias the study
– experimenter attributes
– experimenter expectancies
 effect on experimenter—recording bias
 effect on research participant--
 mediated of expectancies
 handling in animal research

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Construct Validity
(cont'd)
• Ways experimenter may bias the study
– experimenter expectancies
 nonverbal communication in human studies
 magnitude of expectancies—can exist in animal
and human research and can be greater than the
IV

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Internal Validity

• Definition-- accuracy of the inference that

the independent variable caused the effect
observed in the dependent variable
• Primary threat is confounding extraneous
variables
• Eliminate the confounding influence of
extraneous variables by holding their
influence constant

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Internal Validity

• History - any event that occurs between

the pretest and posttest that can produce
the outcome
– differential history occurs in multigroup design
when event has differential impact on groups
• Maturation – internal changes of research
participants that occur over time

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Internal Validity
(cont'd)
• Instrumentation – changes in the
measurement of the dependent variable
– e.g., if human observers change
measurement because they become bored or
fatigued
• Testing – occurs when the influence of
taking the pretest affects the posttest

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Internal Validity
(cont'd)
• Regression Artifact – the tendency for
extreme scores to be closer to average at
posttest
– potential problem if participants with extreme
scores at pretest are selected for study
• Attrition – drop out rate of research
participants
– a potential threat in two group designs where
differential attrition occurs
Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Threats to Internal Validity
(cont'd)
• Selection – potential threat in a two group
design when different selection
procedures are used
• Additive and Interactive effects – produced
by the combined effect of two or more
threats

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 External Validity

• Generalizing across people, settings,

treatment variations, outcomes and times
• A failure to generalize can result from
several factors:
– lack of random selection
– chance variation
– failure to identify interactive effects of
independent variables

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Types of External Validity

• Population validity
– do results generalize from sample to target
population?
• Ecological validity
– do results of study generalize to different
setting?
– common criticism of laboratory experiments

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Types of External Validity
(cont'd)
• Temporal validity
– do results generalize across time?
 seasonal variation
 cyclical variation
• Treatment variation validity
– do results generalize across variations in
treatment?

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
 Types of External Validity
(cont'd)
• Outcome validity
– do results generalize to other, related,
dependent variables?

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
Relationship between Internal
and External Validity
• Relationship between internal and external
validity is often inverse
• Factors that increase our ability to
establish cause and effect tend to
decrease our ability to generalize
• External validity is established through
replication

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.
Relationship between Internal
and External Validity (cont'd)
• Emphasis of internal or external validity
depends on whether or not a causal
relationship has been established

Research Methods, Design, and Analysis, Eleventh Edition Copyright ©2011 by Pearson Education, Inc.
Christensen • Johnson • Turner All rights reserved.