Survey Question Construction Techniques and Tips

Writing effective survey questions

This is an important topic because the structure of your questions has a direct impact on
the usability of the answers. As a result, it should come as no surprise that this topic is so
important there is a whole body of literature devoted to nothing else.

Survey items have two separate parts: the question and the answer. The distinction is
important. A good question not only asks for information clearly, but elicits useful
responses.

Nine rules for writing good questions

Naturally, no question is "good" in all situations, but there are some general rules to
follow. Using these rules and examples will help you write useful questions.

1. Remember your survey's purpose

All other rules and guidelines are based on this one. There was a reason you decided to
spend your time to do your survey, and you should ensure that every question you ask
supports that reason. If you start to get lost while writing your questions, refer back to
this rule.

2. If in doubt, throw it out

This is another way of stating the first rule, but it is important enough to repeat. A
question should never be included in a survey because you can't think of a good reason to
discard it. If you cannot come up with a concrete research benefit that will result from the
question, don't use it.

3. Keep your questions simple

Compound sentences force respondents to keep a lot of information in their heads, and
are likely to produce unpredictable results. Example: "Imagine a situation where the
production supervisor is away from the line, a series of defective parts is being
manufactured, and you just heard that a new client requires ten thousand of these parts in
order to make their production schedule. How empowered do you feel by your
organization to stop the line and make the repairs to the manufacturing equipment?" This
question is too complex for a clear, usable answer. Try breaking it down into component
parts.

4. Stay focused - avoid vague issues

If you ask "When did you last see a movie?" you might get answers that refer to the last
time your respondent rented a video, when you are really interested in the last time the
respondent went out to a movie theater.

Consider too, "Please rate your satisfaction with the service you have received from this
company." This is a fine general question, but will not likely lead to any specific action
steps. Particular elements of service must be probed if responses are to result in specific
recommendations.

5. If a question can be misinterpreted, it will be

"What time do you normally eat dinner?" will be answered differently by people living in
different regions; "dinner" can refer to either the midday or the evening meal. Be clear,
concise, always beware of imprecise language and avoid double negatives.

6. Include only one topic per question (avoid "double-barreled" questions)

How would you interpret the responses to "Please rate your satisfaction with the amount
and kind of care you received while in the hospital." or, a question asking about speed
and accuracy? If you want to be able to come up with specific recommended actions, you
need specific questions.

7. Avoid leading questions

It is easy, and incorrect, to write a question that the respondent believes has a "right"
answer. "Most doctors believe that exercise is good for you. Do you agree?" is an
example of a leading question. Even the most well-meaning researcher can slant results
by including extraneous information in a question. Leading questions can be used to
prejudice results.

8. Consider alternate ways to ask sensitive questions

Some questions are obviously sensitive. Income, drug or alcohol consumption and sexual
habits are clear examples of topics that must be asked about carefully. The question: "Did
you vote in the last election?" has an element of sensitivity in it as well. Respondents
might be unwilling to admit that they did not vote, because of civic pride or
embarrassment. To avoid respondent alienation, it can be useful to mitigate the cost of
answering "No" by including a way out. For example: "There are many reasons why
people don't get a chance to vote. Sometimes they have an emergency, or are ill, or
simply can't get to the polls. Did you vote in the last election?"

Also, people are less likely to lie about their age in face-to-face interviews if they are
asked what year they were born, rather than how old they are.

9. Make sure the respondent has enough information

Asking respondents "How effective has this company's new distribution program been?"
may not be as effective as "Recently, we implemented a new, centralized distribution
system. Did you know this?" Followed by "Have you seen any positive benefits resulting
from this change?" It can be beneficial to break down questions that require background
information into two parts: a screening item describing the situation which asks if the
respondent knows about it, and a follow-up question addressing attitudes the respondent
has about the topic.
Five rules for obtaining usable answers

Useful answers are just as important as good questions. Here are some rules:

1. Response options need to be mutually exclusive and exhaustive

This is the most important rule to follow when providing response options. If response
options are not mutually exclusive, the respondent will have more than one legitimate
place for their answer. The response choices, "1 to 2," "2 to 3" and "More than 3" pose a
problem for someone whose answer is "2."

You must also ensure that the response options you provide cover every possibility.
Asking "Which of the following beverages did you drink at least once during the past
seven days?" and providing a list of coffee, soda and tea might be sufficient if you were
doing a study on the consumption of caffeinated drinks. But, they would not work if you
wanted to know about broader consumption habits. If you are unable to provide a
complete list of options, at least provide an "Other" choice. If the list of choices is too
long, an open ended-question might be a better option.

2. Keep open-ended questions to a minimum

While open-ended (or verbatim) questions are a valuable tool, they should not be over-
used. Not only can they result in respondent fatigue, but they pose problems in terms of
coding and analysis.

3. People interpret things differently, particularly when it comes to time

Trouble-spots include responses such as "Always," "Sometimes" and "Never." You must
build in a temporal frame of reference to ensure that all respondents are answering in the
same way. As in this example from an interviewer-administered questionnaire, "I am
going to read a list of publications. For each one, please tell me whether you read it
regularly. By regularly I mean, at least three out of every four issues."

4. Consider a "Don't Know" response

It is useful to allow people to say they simply do not have an opinion about a topic.
However, some investigators worry that people will opt for that choice, reducing the
ability to analyze responses. Evidence shows that this fear is largely unfounded. The goal
of your research should help you decide if a "Don't Know" option would be wise. For
example, if you only want information from those with an informed opinion or higher
interest, offer a "Don't Know" choice.

5. Provide a meaningful scale

The end points of response scales must be anchored with meaningful labels. For example,
"Please rate your satisfaction with customer service. Lets use a scale where 5 means
'Very Satisfied' and 1 means 'Very Dissatisfied.'" You could also give each point on the
scale a label.
The number of scale points (3, 5 or 7) can have little effect on the conclusions you draw
later. Choosing how many points, then, is often a matter of taste. There are three things to
remember when constructing a response scale. First, an odd number of points provides a
middle alternative. This is a good way to provide respondents with moderate opinions a
way out (similar to the "Dont Know," choice above). Secondly, if measuring extreme
opinions is critical, use a scale with a greater number of points. Finally, you generally
gain nothing by having a scale with more than 7 points and will probably find that you
will collapse larger scales when it comes time to analyze the data.

The price of poorly written questions

Well-written questions are critical. Participants must stay interested. If your respondents
start to feel alienated by threatening, emotional or difficult questions, response rates are
likely to go down and response bias will probably go up.

Also, respondents can get frustrated if your questions do not provide answer choices that
match their opinions or experiences. The quality of your collected data will suffer; your
analyses will be less meaningful; and the whole research process may prove useless or
harmful. So think carefully about the questions you write, look at reputable examples of
questions, and refer to the rules above. If you follow these guidelines, you'll do fine.
Guidelines for creating better questionnaires

Why is good question ordering and layout important? Because the cost of poorly
designed questionnaires is the same as the cost of poorly written questions or a bad
sample: the results you get can be biased, uninterruptable, insufficient or misleading.

This column concentrates on 15 guidelines for better survey design. As with much of
survey research, these guidelines follow common sense rather than hard-and-fast
principles. They are easy to forget, however, because they are so simple. But, keeping
these guidelines in mind will greatly improve your chances of success.

Remember, responding to a survey should be an interesting, stress-free experience. If

respondents become bored, confused or irritated, the results you get back may be of little
use.

Think about the order and layout of the survey as if you had to fill it out. Is the survey
attractive and professional looking? Is it easy to read? Are the questions interesting? Are
there many questions on the same subject or are the topics varied?

Self-administered surveys require respondents to be both interested in and capable of

filling in the survey. In other words, the questionnaire not only has to motivate recipients
in some way through its importance, appearance and questions, but be easy enough to
complete to minimize fatigue, boredom or confusion.

General guidelines for question ordering

1. The first several questions should:

o be easy for respondents to understand
o be important to the study's purpose
o engage the attention and interest of your respondent

Do not begin with an open-ended question or one which respondents might feel
has a "right" answer.

2. It is important to get your respondent interested in the survey at the very

beginning. In cases where the topic is already of interest or importance to the
respondent, start with general questions, then funnel to more specific ones. If the
topic is of low importance to respondents, start with specific questions. This gives
respondents a frame of reference; then ask broader, more general questions.
3. Group questions in sections, and position sections or questions in a logical order.
4. Introduce new sections with a sentence or phrase so that participants have a
chance to switch mental gears.
5. Place questions about sensitive issues such as income, sexual habits or drug abuse
toward the end of the document, or section. This helps avoid alienating, taxing or
in other ways worrying participants.
 6. Consider lists of similar items carefully. For example, you might ask about
product satisfaction using 10 different attributes (price, availability, delivery,
color, fit and so on). The first items on a list such as this usually get treated
differently by respondents than the last items do. This is called position bias.

You can randomly or alphabetically order list items and indicate in the
instructions how they are ordered. This reduces the likelihood that respondents
will see the first items as most important.

7. Put demographic questions at the end of the questionnaire, if possible. There are
at least two reasons for this. First, some demographic questions such as age and
income can be sensitive and should be placed at the end, as discussed in guideline
number 5.

Second, it is better to keep respondents' minds on the purpose of the survey at the
beginning, while you have their attention. Demographic questions rarely require
much thought, so wait until the end when respondents might be tired.

8. Try to minimize the number of times the respondent or the interviewer has to
follow a skip rule. For example "If the answer to question 3 is 'None of the
Above,' skip to question 6." When there are too many skips or when skips become
too complicated, you run the risk of introducing error and confusion.

General guidelines for questionnaire layout

1. Create professional, attractive and uncluttered questionnaires; fonts should be

large enough to avoid eye strain; instructions for completing the survey should be
easy to understand. If your survey has multiple pages, it should be bound in a
booklet so that pages cannot get mixed up. And, if respondents are to see the
survey, it should have an attractive cover.
2. Make the survey easy to complete; the check boxes or lines easy to see; and the
numbers to be circled far enough apart so the respondent or interviewer will not
inadvertently circle two numbers. If you are using scannable forms, where
bubbles need to be filled in completely and with a specific writing utensil such as
a number 2 pencil, make sure the instructions are clear and easy to find.
3. Number your questions clearly. This will lessen the chance, particularly in longer
surveys, of respondents or interviewers getting lost.
4. Start with a brief introduction describing the survey's purpose, the topics being
covered and how the results will be used. Also, mention any incentive for
completing the survey, such as a drawing entry, the opportunity to have a copy of
the results and so on.
5. Leave plenty of room for respondents to write answers to open-ended questions.
Do not supply lines because this could constrain any comments.