RESEARCH 1 Notes

Questionnaire Design | differences between the population and

Methods, Question Types & the sample, but minimizing these will help
you avoid several types of research bias,
Examples including sampling bias, ascertainment
A questionnaire is a list of questions or bias, and under coverage bias.
items used to gather data from
respondents about their attitudes,
experiences, or opinions. Questionnaires Questionnaire methods
can be used to
collect quantitative and/or qualitative infor Questionnaires can be self-
mation. administered, or researcher
administered. Self-administered
questionnaires are more common because
Questionnaires are commonly used in
they are easy to implement and
market research as well as in the social
inexpensive, but researcher-administered
and health sciences. For example, a
questionnaires allow deeper insights.
company may ask for feedback about a
recent customer service experience, or
psychology researchers may investigate Self-administered questionnaires
health risk perceptions using Self-administered questionnaires can be
questionnaires. delivered online or in paper-and-pen
formats, in person or through mail. All
Questionnaires vs. surveys questions are standardized so that all
respondents receive the same questions
A survey is a research method where you with identical wording.
collect and analyze data from a group of
people. A questionnaire is a specific tool or Self-administered questionnaires can be:
instrument for collecting the data.
 cost-effective
Designing a questionnaire means creating  easy to administer for small and
large groups.
valid and reliable questions that address
 anonymous and suitable for
your research objectives, placing them in
sensitive topics
a useful order, and selecting an  self-paced
appropriate method for administration.
But they may also be:
But designing a questionnaire is only one
component of survey research. Survey  unsuitable for people with limited
research also involves defining the literacy or verbal skills
population you’re interested in, choosing  susceptible to a nonresponse
bias (most people invited may not
an appropriate sampling method,
complete the questionnaire)
administering questionnaires, data
 biased towards people
cleansing and analysis, and interpretation. who volunteer because impersonal
survey requests often go ignored.
Sampling is important in survey research
because you’ll often aim to generalize Researcher-administered
your results to the population. Gather data questionnaires
from a sample that represents the range Researcher-administered questionnaires
of views in the population for externally are interviews that take place by phone,
valid results. There will always be some

Questionnaire Research 1
 RESEARCH 1 Notes

in-person, or online between researchers

and respondents.
Categorical variables can be nominal or
Researcher-administered questionnaires ordinal. Quantitative variables can be
can: interval or ratio. Understanding the type of
variable and level of measurement means
 help you ensure the respondents are you can perform appropriate statistical
representative of your target audience. analyses for generalizable results.
 allow clarifications of ambiguous or
unclear questions and answers
 have high response rates because it’s Examples of closed-ended questions
harder to refuse an interview when for different variables
personal attention is given to
Nominal variables include categories
respondents.
that can’t be ranked, such as race or
ethnicity. This includes binary or
But researcher-administered
dichotomous categories.
questionnaires can be limiting in terms of
resources. They are: It’s best to include categories that cover
all possible answers and are mutually
 costly and time-consuming to perform. exclusive. There should be no overlap
 more difficult to analyze if you have between response items.
qualitative responses.
 likely to contain experimenter bias or In binary or dichotomous questions, you’ll
demand characteristics. give respondents only two options to
 likely to encourage social desirability choose from.
bias in responses because of a lack of
Example: Nominal variables
anonymity.

Open-ended vs. closed-ended

questions.

Your questionnaire can include open-

ended or closed-ended questions or a
combination of both.

Using closed-ended questions limits your

responses, while open-ended questions
enable a broad range of answers. You’ll
need to balance these considerations with
your available time and resources.

Closed-ended questions.

Closed-ended, or restricted-choice,
questions offer respondents a fixed set of
choices to select from. Closed-ended
questions are best for collecting data on
categorical or quantitative variables.

Questionnaire Research 1
 RESEARCH 1 Notes

Ordinal variables include categories that quickly. However, you might still miss
can be ranked. Consider how wide or important answers that are relevant to
narrow a range you’ll include in your respondents. An incomplete set of
response items, and their relevance to response items may force some
your respondents. respondents to pick the closest alternative
to their true answer. These types of
questions may also miss out on valuable
detail.

To solve these problems, you can make

questions partially closed-ended, and
include an open-ended option where
respondents can fill in their own answer.

Open-ended questions
Open-ended, or long-form, questions allow
respondents to give answers in their own
words. Because there are no restrictions
on their choices, respondents can answer
in ways that researchers may not have
otherwise considered. For example,
respondents may want to answer
Likert scale questions collect ordinal data
“multiracial” for the question on race
using rating scales with 5 or 7 points. rather than selecting from a restricted list.

Example: Open-ended questions

When you have four or more Likert-type

questions, you can treat the composite data as
quantitative data on an interval scale.
Open-ended questions have a few
Intelligence tests, psychological scales, and
personality inventories use multiple Likert-type
downsides.
questions to collect interval data. They require more time and effort from
With interval or ratio scales, you can apply respondents, which may deter them
strong statistical hypothesis tests to address from completing the questionnaire.
your research aims.
For researchers, understanding and
Pros and cons of closed-ended summarizing responses to these
questions questions can take a lot of time and
resources. You’ll need to develop a
Well-designed closed-ended questions are systematic coding scheme to
easy to understand and can be answered categorize answers, and you may also

Questionnaire Research 1
 RESEARCH 1 Notes

need to involve other researchers in framed in a particular direction. It’s best

data analysis for high reliability. practice to provide a counter argument
within the question as well.
Question wording
Question wording can influence your
respondents’ answers, especially if the
language is unclear, ambiguous, or biased.
Good questions need to be understood by
all respondents in the same way (reliable)
and measure exactly what you’re
interested in (valid).

Use clear language.

Avoid leading questions.
You should design questions with your
target audience in mind. Consider their Leading questions guide respondents
familiarity with your questionnaire topics towards answering in specific ways, even
and language and tailor your questions to if that’s not how they truly feel, by
them. explicitly or implicitly providing them
with extra information.
For readability and clarity, avoid jargon or
overly complex language. Don’t use double
negatives because they can be harder to
It’s best to keep your questions short and
understand.
specific to your topic of interest.
Use balanced framing.

Respondents often answer in different ways

Example: Leading questions
depending on the question framing. Positive
frames are interpreted as more neutral than The average daily work commute in the
negative frames and may encourage more US takes 54.2 minutes and costs $29 per
socially desirable answers. day. Since 2020, working from home has
saved many employees time and money.
Do you favor flexible work-from-home
policies even after it’s safe to return to
offices?

Experts agree that a well-balanced diet

provides sufficient vitamins and
minerals, and multivitamins and
supplements are not necessary or
effective. Do you agree or disagree that
multivitamins are helpful for balanced
Use a mix of both positive and negative nutrition?
frames to avoid research bias and ensure
that your question wording is balanced Keep your questions focused.
wherever possible. Ask about only one idea at a time and
Unbalanced questions focus on only one avoid double-barreled questions. Double-
barreled questions ask about more than
side of an argument. Respondents may be
less likely to oppose the question if it is

Questionnaire Research 1
 RESEARCH 1 Notes

one item at a time, which can confuse randomize the question order between
respondents. respondents.

Example: Double-barreled question

Logical flow

Using a logical flow to your question

order means starting with simple
questions, such as behavioral or opinion
questions, and ending with more
complex, sensitive, or controversial
This question could be difficult to answer questions.
for respondents who feel strongly about The question order that you use can
the right to clean drinking water but not significantly affect the responses by
high-speed internet. They might only priming them in specific directions.
answer about the topic they feel Question order effects, or context effects,
passionate about or provide a neutral occur when earlier questions influence
answer instead – but neither of these the responses to later questions,
options capture their true answers. reducing the validity of your
Instead, you should ask two separate questionnaire.
questions to gauge respondents’ While demographic questions are usually
opinions. unaffected by order effects, questions
about opinions and attitudes are more
susceptible to them.

Example: Order effects

Presidential approval ratings are often

influenced by any related previous
questions or references to political
situations. These ratings are lower if they
follow relevant questions.
How knowledgeable are you about Joe
Biden’s executive orders in his first 100
days?
Are you satisfied or dissatisfied with the
way Joe Biden is managing the economy?
Do you approve or disapprove of the way
Joe Biden is handling his job as
president?
For this reason, presidential approval
Question order
ratings questions are asked at the start
You can organize the questions logically, of surveys to measure opinions more
with a clear progression from simple to accurately.
complex. Alternatively, you can

Questionnaire Research 1
 RESEARCH 1 Notes

It’s important to minimize order effects Are the respondents familiar with the
because they can be a source of language and terms used in your questions?
systematic error or bias in your study. Would any of the questions insult, confuse, or
embarrass them?
Do the response items for any closed-ended
questions capture all possible answers?
Randomization Are the response items mutually exclusive?
Do the respondents have time to respond to
Randomization involves presenting open-ended questions?
individual respondents with the same Consider all possible options for responses to
questionnaire but with different question closed-ended questions. From a respondent’s
orders. perspective, a lack of response options
reflecting their point of view or true answer
may make them feel alienated or excluded. In
When you use randomization, order
turn, they’ll become disengaged or
effects will be minimized in your dataset.
inattentive to the rest of the questionnaire.
But a randomized order may also make it
harder for respondents to process your Step 3: Decide on your questionnaire length
questionnaire. Some questions may need and question order
more cognitive effort, while others are Once you have your questions, make sure
easier to answer, so a random order that the length and order of your questions
could require more time or mental are appropriate for your sample.
capacity for respondents to switch
If respondents are not being incentivized or
between questions.
compensated, keep your questionnaire short
and easy to answer. Otherwise, your sample
Step-by-step guide to design may be biased with only highly motivated
respondents completing the questionnaire.
Step 1: Define your goals and objectives
The first step of designing a questionnaire is Decide on your question order based on your
determining your aims. aims and resources. Use a logical flow if your
respondents have limited time or if you
What topics or experiences are you studying? cannot randomize questions. Randomizing
questions helps you avoid bias, but it can
What specifically do you want to find out? take more complex statistical analysis to
Is a self-report questionnaire an appropriate interpret your data.
tool for investigating this topic?
Once you’ve specified your research aims, Step 4: Pretest your questionnaire.
you can operationalize your variables of When you have a complete list of questions,
interest into questionnaire items. you’ll need to pretest it to make sure what
Operationalizing concepts means turning you’re asking is always clear and
them from abstract ideas into concrete unambiguous. Pretesting helps you catch any
measurements. Every question needs to errors or points of confusion before
address a defined need and have a clear performing your study.
purpose.
Ask friends, classmates, or members of your
Step 2: Use questions that are suitable for target audience to complete your
your sample questionnaire using the same method you’ll
Create appropriate questions by taking the use for your research. Find out if any
perspective of your respondents. Consider questions were particularly difficult to answer
their language proficiency and available time or if the directions were unclear or
and energy when designing your inconsistent and make changes as necessary.
questionnaire.

Questionnaire Research 1
 RESEARCH 1 Notes

If you have the resources, running a pilot

study will help you test the validity and
reliability of your questionnaire. A pilot study
is a practice run of the full study, and it
includes sampling, data collection, and
analysis. You can find out whether your
procedures are unfeasible or susceptible to
bias and make changes in time, but you can’t
test a hypothesis with this type of study
because it’s usually statistically
underpowered.

Questionnaire Research 1