Measurement

Reliability
Validity
Levels of Measurement
Scales
How we figure out what to measure
• Conceptualization
– Process of taking a construct and refining it by
giving it a conceptual or theoretical definition
– Research focusing on college students
• In Ohio? What region? Age? Major?
• Operationalization
– Links a conceptual definition to a specific set of
measurement techniques
Coming up with a measure
• Remember the conceptual definition
• Keep an open mind
• Borrow from others
• Anticipate difficulties
• Don’t forget units of analysis
Empirical Hypothesis
• The degree of
association
• How well
operationalized
variables are associated
(or not) with the
concept construct
determines the
hypothesis
Reliability
• Reliability means dependability or consistency
• Same thing occurs over and over under same
conditions
• A scale, for example
• How dependable is the study?
• Is the study consistent, or does it yield wide varying
results?
• Can the study be replicated?
Reliability
• Measurement directly affects the quality of
conclusions.
• Care is needed to make sure that results are
not corrupted by improper measurement.
• The operational definition of a concept should
have a precise meaning:
– The terms by which you measure a concept
should be explicit.
Reliability
• Reliability and validity are the biggest threats
to proper measurement.
• Reliability is the extent to which an

experiment, test, or any measuring procedure
yields the same results on repeated trials.
• Do you get the same result every time?

Reliability
• Three tests of reliability:
– Test-retest method
• Applying the same test to the same observations after a
period of time and then comparing the results of the
different measurements
– Alternative form method
• Two different measures of the same concept administered
to the same respondents at different times before the scores
are compared
– Split-halves method
• Divide a multi-item measure into two measures with both of
the new measures applied at the same time
Improving Reliability
• Clearly conceptualize constructs
• Increase level of measurement
• Use multiple indicators of a variable
– Triangulation
• Use pretests, pilot studies, and replication
Validity
• A valid measure is one that measures what it
is supposed to measure, in other words, the
degree of correspondence between the
measure and the concept it is thought to
measure.
• Four tests of validity

Validity
• Truthfulness
• Refers to the match between a
construct and a measure
• Want it to be valid for a particular
purpose and definition
• How good is the measure?
• Is the data measured correctly?
• Is the data analyzed correctly
(statistical)?
Internal Validity
• Are there errors as a result of the internal
design of the study?
• Are there errors as a result of the controls?
• Internal validity problems can occur from a

flawed survey along with a multitude of other
factors
External Validity
• Can your experiment’s findings be
generalized?
• External Validity questions are evident in

every study; however, methods exist to keep
external validity high and the number of
external flaws low
Types of Validity
• Face validity
– Judgment that the indicator really measures the construct
• Content validity
– Does your measure represent the full content of a defintion?
• Criterion validity
– Use some standard or criteria to indicate a construct accurately
• Concurrent validity
– Indicator must be associated with a preexisting indicator judged to be
valid
• Predictive validity
– Indicator predicts future events that are logically related to a construct
Types of Validity
• 1) If I create a new test of mathematical ability for high school students and test it by having high school
math teachers look at it and tell me if it seems appropriate, I am measuring for ____________________
validity.
•
• 2) If I am examining an individuals ability to cope with stress and have three attributes I am particularly
interested in and I am checking to see if my construct hits on all three attributes, I am measuring for
_______________________ validity.
•
• 3) If I create a new test for cognitive recognition and students that score high on it also score high on
previously existing tests for cognitive recognition, I have demonstrated ____________________ validity.
•
• 4) If I compare my measure for testing the potential to suffer from childhood diabetes with a previously
used test, I am looking for _________________________validity.
•
• 5) If I create a new test of intelligence and students that score high on it also do better in college than
those who score lowly, I have shown ___________________ validity.
Validity
• Tests of validity are not as good as tests of
reliability.
• Reliability is easy to demonstrate through
some form of repeated trials.
• Validity is more difficult because we can never
be sure about the true value of a concept:
– Especially true with abstract concepts
Validity
• Whereas a valid measure is reliable (because if
truly valid, it will measure the concept
correctly every time), a reliable measure is not
necessarily valid.
• The measure could be measuring the concept

incorrectly in a consistent way.
Relationship between reliability and
validity
• The level of measurement of a variable
describes
– The amount of precision associated with a
variable
– The mathematical properties of the variable
• Both precision and mathematical properties
increase as you increase the level of
measurement from nominal to ratio.
• Continuous v. discrete variables
– Continuous
• Have an infinite number of values or attributes that
flow directly along a continuum
• Temperature, age, income, crime rate
– Discrete
• Relatively fixed set of separate values or attributes
• Gender, religion, marital status
Nominal Level
• Only reports a
difference
• Candidate preference,
religious preference,
Yes/No, etc.
• Discrete Variables
• The level of measurement of a variable
describes
– The amount of precision associated with a
variable
– The mathematical properties of the variable
• Both precision and mathematical properties
increase as you increase the level of
measurement from nominal to ratio.
Ordinal Level
• Rank ordered
• Grades, opinion
• Strongly Agree
• Agree
• Disagree
• Strongly Disagree
• At the ordinal level, categories may be ranked in
order in addition to indicating a difference
between categories.
• Example: Please indicate the highest level of

education you reached (elem., high, college,
more).
• Precision: A little more precision and can be used

with more statistical tools
Interval/Ratio Level
• A specified distance
• Interval does not contain a

true zero point (ratio does)
• Interval: IQ, SAT

• Ratio: years of school,
income
• The interval level includes all of the
information of the preceding levels and adds
meaningful intervals between values of the
variable but does not use a meaningful zero.
• Example: What did you score on the SAT?
• Precision: More precision and can be used

with most statistical tools
• The ratio level adds a meaningful zero to the
interval level.
• Example: How many years of education?
• Precision: Most precision and can be used

with most statistical tools
Scales
• Some concepts can be captured with a single
question.
• More complex concepts may require a multi-
item measure consisting of several questions
that capture different components of the
concept and increase validity.
Scales
• Summation index:
– Combines the scores on multiple questions to create one
single measure of a concept
• Likert scale:
– Uses only select questions from an index that differentiate
between different respondents to create a single score for
each respondent
Scales
• Guttman scale:
– Has answer choices arranged in an ordinal manner;
respondents will agree with each of the lower-ranked
answers if they agree with a higher-ranked answer
• Factor analysis:
– Allows researchers to uncover patterns across related
measures to create summary variables that represent
different dimensions of the same concept
Mutually Exclusive
• “One and only.”
• One may only fit the criteria of one category
• Ex: Religion:
Christian, non-Christian, Jewish, Buddhist
NOT MUTUALLY EXCLUSIVE

Exhaustive
• All cases fit into one category
• Ex: If the Election were today, would you

support Sherrod Brown, the Democrat, or
Mike DeWine, the Republican or neither?
• NOT EXHAUSTIVE
Missing Data
• No survey is perfect, and certain questions will be
left unanswered or completely skipped
• Remedy?
• A “catch all” category and/or a way to factor out

missing data
• Yet, missing data can still mislead a study

One Last Thing
• Feeling Thermometers
• Likert Scales
• Response set problem
– How do we fix this???

Measurement

Uploaded by

Copyright:

Available Formats

Measurement

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Measurement

Uploaded by

Copyright:

Available Formats

Reliability

• Reliability is the extent to which an

• Do you get the same result every time?

• Four tests of validity

• Are there errors as a result of the controls?

• Internal validity problems can occur from a

• External Validity questions are evident in

• The measure could be measuring the concept

• Example: Please indicate the highest level of

• Precision: A little more precision and can be used

• Interval does not contain a

• Interval: IQ, SAT

• Example: What did you score on the SAT?

• Precision: More precision and can be used

• Example: How many years of education?

• Precision: Most precision and can be used

NOT MUTUALLY EXCLUSIVE

• Ex: If the Election were today, would you

• A “catch all” category and/or a way to factor out

• Yet, missing data can still mislead a study

You might also like