SURVEY

MIDTERM SCHEDULE
October 26: Theory and RQ; Review day
October 31: Midterm, Veritas Hall B207, bring computers
(further instructions in two weeks)
GROUP MEMBER EVALUATIONS
ASSIGNMENTS
Group submission: Proposal part 1 due on October 17 before class.

October 31: midterm

Q&A
In class, we learned that internal validity and external validity have an
inverse relationship. Is there any possible way to achieve both? Also, if it's
difficult to achieve both types of validity when conducting an experimental
research, which one should we give priority to?
A causal relationship can be tested in an artificial lab setting or in the ‘real world’. A
lab setting ensures higher internal validity because external influences can be
minimized. However, the external validity diminishes because a lab environment is
different than the ‘outside world’ (that does have external influencing factors). -
scribbr
THREATS TO EXTERNAL VALIDITY

Sample characteristics
Low response rate
Mortality
Situational factors
Social desirability or memory recall bias
“External validity is the cornerstone of a good experiment design, but is a bit difficult
achieve when using well designed experimental study.”

“Studies are high in external validity to the extent that the result can be generalized
to people and situations beyond those actually studied. Although experiments can
seem “artificial”—and low in external validity—it is important to consider whether
the psychological processes under study are likely to operate in other people and
situations.”

Example: difficult to generalize to very large target populations, controlled

environments, etc.
IT’S ABOUT CONFIDENCE AND YOUR DISCUSSION!
“The tradeoff is reframed as a tradeoff between the degree of confidence that a
researcher can attain about the internal validity of her result and the degree of
confidence she can attain regarding its external validity.”

So tradeoff is less between internal validity and external validity, but more between
the degrees of confidence you have with respect to having the former two.

It’s about the knowledge we gain from your study and your study can contribute
to further research and knowing your limitations!
BALANCING EXTERNAL AND INTERNAL VALIDITY
A solution to this trade-off is to conduct the research first in a controlled (artificial)
environment to establish the existence of a causal relationship, followed by a field
experiment to analyze if the results hold in the real world. –scribbr

Choosing topics that are feasible to study...best case scenario: your experiment mimics
real-life circumstances, but a tradeoff will always exist.
If conducting experiments, if you have low internal validity, would you also have low external validity?

“...it does not make much sense to ask whether a result is valid outside the experimental circumstances unless we are confident
that it does therein ” (Guala 2003, 1198).

“It could be the case, for example, that an experimental result must be internally valid to some degree for external validity to
be achievable.”

“Lack of internal validity implies that the results of the study deviate from the truth, and, therefore, we cannot draw any
conclusions; hence, if the results of a trial are not internally valid, external validity is irrelevant.”

“While the conflict between the prerequisite and tradeoff views has been documented in the literature (see e.g. Persson and
Wallin 2015 and Jimenez-Buedo and Miller 2010), there have been few attempts to resolve it.”
 Mini-review game #3
Defining surveys
Survey types
Developing questions
AGENDA Good and bad practices
Question types
KAHOOT
DEFINING CHARACTERISTICS OF SURVEYS
Data collection method!
Descriptive (or exploratory) in purpose
 surveys can be further used for explanatory research and analysis methods

Data obtained by asking questions

 Questionnaires as the primary instrument
 Close-ended vs open-ended

Target content
 Behavior, attitudes/beliefs/opinions, characteristics, expectations, self-classification, knowledge, self-
reported activities
RESEARCH DESIGN/SURVEY TYPES
Cross-sectional
Surveys
Longitudinal (experiments or observational)
RESEARCH STRUCTURE
Sampling Data collection
Overall design method
Cross- SRS Survey
sectional Stratified Observation
Longitudinal random Interviews
including sampling (qual.)
Exp. Cluster
Trend study random
Panel study sampling
Etc. Systematic
sampling
Etc.
CROSS-SECTIONAL
Single snapshot of social phenomena
Used to investigate a sample drawn from a
predetermined population
Descriptive, explanatory, or exploratory
Compare groups or individuals in the sample
LONGITUDINAL
• Over time, repeated measures
• Descriptive, explanatory, or exploratory
• Different types
• Trend studies
• Panel studies
• Cohort studies
• Experimental designs
TREND STUDIES (TIME-SERIES)
Trend studies involve surveying different samples from a population whose members
may change at different points of time

Different groups compared over time

Group1 Group 2 Group 3

Time 1 Time 2 Time 3

PANEL STUDY
The researcher surveys the same sample of individuals at different times during the
course of the survey
COHORT STUDY
Cohort studies refer to an approach that follows research participants over a period of time (often many
years). Specifically, cohort studies recruit and follow participants who share a common characteristic, such as a
particular occupation or demographic similarity (Barret & Noble, 2019)

• Cohort studies are observational

• Share a common characteristic, such as a particular occupation or demographic similarity

• The sample must not present with the disease under study at the outset of the study

• Mostly used in the medical sciences, but also found in the applied social sciences

• Cohort studies typically gather data using methods of observation

• Conducted where controlled experiments are unethical

COHORT STUDY WITH TWO GROUPS

Condition present
Exposed to
some risk
Condition absent
Compare rates

Condition present
Not
exposed
Condition absent

Time
EXPERIMENTS
Determine causality
Manipulate research conditions/IV
Internal validity
STEPS WHEN CONDUCTING SURVEYS

Read literature Develop Identify target

Define problem and develop Decide on survey questionnaire after Choose mode of Collect/measure Analyze data Writeup
research question design type conceptualization population and data collection data
sample
and hypotheses of variables
QUESTIONNAIRE
A questionnaire is a research instrument that consists of a set of questions
to collect information from a respondent
 Self-administered
 Mostly close-ended (multiple choice) in quantitative research
 Measuring opinions, attitudes, beliefs, self-reported activities, etc.
 Can be used for both qualitative and quantitative research, mostly quantitative
 Open vs close-ended questions
GOOD PRACTICES IN
DEVELOPING QUESTIONNAIRES
Avoid jargon, slang, and abbreviations

Avoid ambiguity, confusion, and vagueness

Avoid emotional and prestige bias

Avoid double-barreled questions

Avoid questions beyond respondents’ capabilities

Avoid false premises

Avoid leading questions
Avoid asking about distant future intentions

Avoid double negatives

Avoid overlapping or unbalanced response categories

AVOID AMBIGUITY, CONFUSION, AND VAGUENESS

Do you typically prefer studying by yourself?

How many times did you drink last year?

AVOID JARGON, SLANG, AND ADVANCED
VOCABULARY
Do you like chilling with your friends?

Are your ancestors primarily Anglo-Saxon?

Do you consider yourself itinerant?

AVOID DOUBLE-BARRELED QUESTIONS
A double-barreled question consists of two or more questions mixed together

Do you like reading and writing?

How many times do you read in a week, and what genre of books do you prefer?
WHAT’S WRONG WITH THIS
QUESTION?
Should the mayor allocate funds to fix streets with
large potholes that have become dangerous and
are forcing drivers to make costly repairs?
AVOID LEADING QUESTIONS
A leading (or loaded) question is one that leads the respondent to one response over
another by its wording

Should the mayor allocate funds to fix streets with large potholes that have
become dangerous and are forcing drivers to make costly repairs?
CHALLENGES LINKED TO
RESPONDENTS
Respondent recall bias
Social desirability/prestige bias
Neutral position, selective bias
ACTIVITY: OUR MEMORY
FALSE MEMORY STATS
DNA evidence used to overturn over 300 convictions since 1980s. 75% of these
convictions based on eye witness testimony.
RESPONDENT RECALL BIAS
Refers to the inaccurate or incomplete recollection of events by the respondent

Memory is not as trustworthy as we are led to believe...so we must be careful when

asking question relating to:
 Simultaneous events
 Back-to-back events

Telescoping refers to recalling an event earlier or later than when it actually

occurred
ADDRESSING TELESCOPING
Situational framing
 Tell me what happened on the day you took your law test, starting with the morning? Evening?

Decomposition
 Did you go shopping last month?
 The first week of the month?
 The second week of the month?

Landmark anchoring
 Were you married before the president was elected?

Bounded recall (for panel studies)

 How many jobs have you had since we last talked, a year ago?
SOCIAL DESIRABILITY BIAS
AND SENSITIVE QUESTIONS
Refers to the tendency of survey respondents to
answer questions in a manner that will be viewed
favorably by others

We must be careful to address questions that are

sensitive
 Drug or alcohol use, criminal behavior, socially unpopular behaviors, mental
health problems, etc.
ADDRESSING SOCIAL DESIRABILITY AND
SENSITIVE QUESTIONS
Strategies to address dishonesty in answering
 Create comfort and trust
 Enhanced phrasing
 Establish a desensitizing context
 Use anonymous questioning methods
CREATE COMFORT AND TRUST
1. What is your age? 1. How many alcoholic beverages
(glass of wine, can of beer, shot of
2. How much do you like sunny days? liquor) do you drink per week?
Scale: 5 (love them) – 1 (hate them)
1 2 3 4 5 2. What is your age?
3. How many alcoholic beverages 3. How much do you like sunny days?
(glass of wine, can of beer, shot of Scale: 5 (love them) – 1 (hate them)
liquor) do you drink per week? 1 2 3 4 5
 ORDER EFFECT/CONTEXT EFFECT
Refers to the phenomenon that question sequence may influence respondents’ answers in a more or less systematic
fashion (Schuman & Presser, 1981)

As cited in Neuman
NEUMAN AND NEUMAN

“Such order effects appear to be

strongest for people who lack strong
views, for less educated respondents,
and for older respondents or those
with memory loss”
ORGANIZATION OF OVERALL INSTRUMENT: USING
FUNNEL SEQUENCE
A questionnaire has context questions (opening), middle and ending questions
 Opening questions
 “This questionnaire is divided into the following sections”
 Example question: How many members are in your immediate family (family members you live with)?

 Middle questions
 “Now I would like to ask you questions about your family context”
 Example question: In any given week, how many times do you have arguments with your immediate family members?
 “In the following section, you will be asked question on peer interaction in the school setting”
 Example question: Please rate the following statement. My peers help me understand school lessons.

 Ending questions
 Less sensitive questions
 Always say thank you
ENHANCED PHRASING
Have you ever hurt someone in a fight?

Have you ever used self-defense in a fight?

Have you ever resorted to physical altercations?
ESTABLISH A DESENSITIZING CONTEXT
1. Have you ever operated a vehicle while driving under the influence?
2. How many alcoholic beverages (glass of wine, can of beer, shot of liquor) do
you drink per week?
ANONYMOUS QUESTIONING METHOD
SOCIAL DESIRABILITY BIAS AND KNOWLEDGE-
BASED QUESTIONS
Knowledge questions are questions that ask about facts

Addressing dishonesty in answering knowledge questions

 Sleeper questions
 Probing
SLEEPER QUESTIONS
A sleeper question is about nonexistent people or events

https://sociologyinfocus.com/jimmy-kimmel-starbucks-and-
sleeper-questions-2-2/
PROBING
A follow-up question designed to encourage deeper thinking about a specific topic
and obtain more detailed response
 Do you consider yourself an informed citizen on politics? Yes or no
 What is a key issue that you have a strong opinion on?
 What is your opinion on x y z?
NEUTRAL POSITIONS, FLOATERS,
AND SELECTIVE REFUSALS
Neutral position or fence-sitters choose the “no opinion” or “not
applicable” choice because it is easiest (satisficing)
 On average, giving a “no opinion” option increases the chance
that respondents will choose “no opinion” by 20 percent (Schuman
& Presser, 1981)

Selective refusals skip questions that they do not want to answer

Floaters choose at random , and choose “no opinion” when

available
HOW TO ADDRESS NONVALID RESPONSES
Remove people who choose “no opinion” and/or refuse to answer a question
 What is the key disadvantage of doing this?

Adding open-ended questions can help identify floaters (Smith, 1984)

Adding probing questions can address floaters as well
Using advanced statistical methods
TYPICAL QUESTION TYPES IN QUANTITATIVE
QUESTIONNAIRES
Open-ended questions
 What is your age?
 What is your opinion on COVID-19 restrictions on social interaction?

Close-ended questions
 Did you find what you were looking for today? YES or NO
 How often do use public transportation in any given week? I don’t use public transportation, Once a
week, More than once a week, Everyday

Scales
SCALES
Scaling measures the intensity, direction, level, or potency of a variable, and is an
ordered series of response options, presented verbally or numerically from which the
respondents select to indicate their level of feeling about the measured attribute
Likert-scale: A type of rating scale used to measure attitudes or opinions. With this
scale, respondents are asked to rate items on a level of agreement
 There are equal numbers of positive and negative options on the scale
 Has a central neutral option, which means there will always be an odd set of answers
available to the respondent
 Measures ordinal data
SCALED RESPONSES TO VARIABLES AND
STATISTICAL ANALYSIS

Respondent A: 5
Respondent B: 5
Respondent C: 2
On average, respondents agree that they participate in science
courses to get a good grade.
SCALES AND INDICES

Question 1 Question 2 Science Motivation Index

Respondent A: 5 Respondent A: 3 Respondent A: 8
Respondent B: 5 Respondent B: 5 Respondent B: 10
Respondent C: 2 Respondent C: 4 Respondent C: 6
MATRIX QUESTIONS
Matrix question A survey research inquiry that groups together a set of questions that share the same
answer categories in a compact form. (Neuman)
FILTER QUESTIONS
 Quasi-filter: A survey research inquiry that includes the answer choice “no opinion,” “unsure,” or “don’t
know”
 Full-filter: A survey research inquiry that first asks respondents whether they have an opinion or know
about a topic; then only those with an opinion or knowledge are asked specifically about the topic in a
follow-up question or follow-up questions
 Do you have an opinion on the keystone pipeline?
 Please rate this statement: The keystone pipeline is a detriment to the environment.
OPEN- VS CLOSE-ENDED QUESTIONS

Closed Open
Advantages • Good fit for coding and quant • Permit a diversity of answers
analysis • More detailed responses
• Good for sensitive questions • Good for complex issues
• Replication is easier
• Easier to answer
Disadvantages • More fence-sitters and • Reponses may be irrelevant
floaters • Coding and statistical analysis
• Clerical mistakes becomes challenging, if not
• Force answers impossible
• Respondents may not finish

Neuman, 2014
ETHICS IN SURVEYS
Invasion of privacy
 Costs and benefits of the survey

Voluntary participation in taking the survey

 Explicitly stated in the beginning and informed consent

Voluntary participation in answering any question

 Explicitly stated in the beginning and informed consent

Pseudosurvey
 A false and deceptive survey-like instrument using the format of a survey interview but whose true
purpose is to persuade a respondent
QUESTIONNAIRE DESIGN ELEMENTS
Length of Survey or Questionnaire
What questions should I ask?
 Background
 Questions about independent and dependent variables
 Questions about extraneous variables
 Any question relating to a variable that may have an impact on the study (a review of literature can
help you identify these variables)
FACTORS AFFECTING RESPONSE RATE
Location: Could a sampled respondent be located?
Contact: Was a located respondent at home or reached after many attempts?
Eligibility: Was the contacted respondent the proper age, race, gender, citizenship,
and so on for the survey purpose?
Cooperation: Was an eligible respondent willing to be interviewed or fill in a
questionnaire?
Completion: Did a cooperating respondent stop answering before the end or start
answering most questions with “do not know” or “no opinion”?
Total response rate: The product of all of the individual rates relating to location,
contact, eligibility, cooperation, and completion.
TOTAL RESPONSE RATE
Wendy has a sampling frame of 1000 people. She draws a representative sample of 250
people from the sampling frame. She is able to locate the contact information of 200 of the
sample drawn. She calls 200, but only makes contact with 190. Of the 190, she determines
that 140 are eligible for her study. Among the 140, she finds that 110 cooperate and are
willing to participate. Among the 110, 80 complete the questionnaire.
Location: .80
Contact: .95
Eligibility: .73
Cooperation: .78
Completion: .73
Total response rate = .80 X .95 X .73 X .78 X .73 = .32 or 32 percent
WAYS OF ADMINISTERING SURVEYS
Mail and Self-Administered Questionnaires (lowest response rate, cheap, slowest)
Telephone Interviews (moderate response rate, more expensive, fast)
Web Surveys (moderate response rate, cheapest, fastest)
Face-to-face (highest response rate, most expensive, slow)
Low response rate is a threat to what?
RESPONSE RATE AND REPRESENTATIVENESS
Response rate is important IF it bears on [has an effect on] [external validity] (Cook,
Heath, & Thompson, 2000)

What are some factors relating to the sample that render response rate important?
QUESTIONS?