Unit 1: Data Analysis

Page 1 of 17
Lesson: Data Collection

DEFINITIONS:
QUALITATIVE SIMILAR SAMPLE LONGITUDINAL CENSUS
VARIABLE STUDY
QUANTITATIVE
ENTIRE POPULATION ANALYZING VARIABLE POPULATION
CROSS-
SECTIONAL
STUDY

________________________________ All individual (people, places, things, etc...) belonging to the group of being
studied.

________________________________ A section of individual taken from a population.

________________________________ Survey the ENTIRE population.

________________________________ ________________________________ Variables that can be measured numerically.

________________________________ ________________________________ Variables that cannot be measured numerically

and are categorical nature.

________________________________________________________________ A study that considers individuals from different

groups at the same time.

________________________________________________________________ A study that considers individuals over a

prolonged period of time.

What makes a good sample?

Ideally, a sample is a good representation of the ____________________________ population. This means that the

results you get from analyzing the sample should be ____________________________ to the results you would

have gotten from ____________________________ the entire ____________________________.

Page 2 of 17
Practice for Data Collection:
For each of the following
i) identify the population
ii) Provide a possible Quantitative/Qualitative variable that can be used for the research
iii) determine if the study should be longitudinal or cross-sectional

1) Conducting a poll for the voters’ choice on the next Ontario election
Population: ________________________________
Type of Variable: ________________________________
Type of Study: ________________________________

2) Preference on wearing ordinary glasses or contact lenses for women

Population: ________________________________
Type of Variable: ________________________________
Type of Study: ________________________________

3) Comparison on depreciation rate for domestic and foreign cars in North America for the first year
Population: ________________________________
Type of Variable: ________________________________
Type of Study: ________________________________

4) The effect of breakfast on high school students’ academic performance

Population: ________________________________
Type of Variable: ________________________________
Type of Study: ________________________________

5) Which gender would more likely be getting speeding tickets in the age between 17-21?
Population: ________________________________
Type of Variable: ________________________________
Type of Study: ________________________________

Page 3 of 17
Lesson: Types of Data

Data gathered directly by

the researcher in the act of ______________________ Data
conducting research or an gathered by someone other
experiment. Data can be than the researcher.
gathered by surveys or
through experimentation.
__
Da ____
gro ta th ___
u a __
_ the ped t is o ___
__ su suc rgan ___
___ gh pe m ov h as ized __
_ ou
__
___ d thr n rio
d o er a findi or
_ ere tio r ti give ng __
__ __
___ gath enta me n _
_ y
_ ta rim __ t”
Da expe _ __ red b bjec
_ _ he “s u
__ t
___ ta ga of the
_ Da n
rv atio
se
ob

an
t hat c c
a ta cifi
t s of d by spe ___________________ variable
sis d
Con groupe gories. has a natural ordering of its
be cate possible values, but the distances
between the values are undefined.

_______________ Type of
categorical variable that describes
a name, label, or category with no
natural order.
_____
_____
__
____
th
num at des _____ va
eric c r
ally ribes a iable
mea
valu sure ______________________ A numeric
e d
variable which can assume an
infinite number of real values

________________________ A numeric
variable that takes only a finite
number of real values

Page 4 of 17
Practice: Types of Data
1. Use the words given to complete the sentences below.

Discrete, Continuous, Categorical, Secondary

32 pupils in a classroom fill out surveys….

a) Their favourite type of fruit is ____________ data.

b) The span of the pupils hands is _____________data.
c) The number of siblings each pupil has is _________data.

2. Data is collected on the following areas. For each, identify all variables in the sentence as: continuous,
discrete, or qualitative. The data may fit more than one of these categories.
a) Favourite colours of 30 students _____________________
b) Time taken to run 100 m (to the nearest second) _____________________
c) Duration of journey to school _____________________
d) Which subjects were chosen as an option _____________________
e) Number of students in the school each year _____________________
f) Cost of the journey to school _____________________

3. Sarah is reading an energy saving article on the internet. The article covers a broad range of topics. She
learns that the average UK household uses 349.44254 litres of water per day.
a) Is household usage of water discrete or continuous data? _____________________
b) Is Sarah working with primary or secondary data? _____________________

4. Give one advantage and one disadvantage of using secondary data. _____________________

5. Which of the following is a significant problem specifically associated with secondary data?
a) It is not be accurate.
b) It is biased.
c) It may not address the aims.
d) It is expensive.

Answers:
1.a)categorical b)continuous c)discrete 2.qualitative for colours, discrete for # of students b)discrete for rounded time recorded c)
continuous for duration and continuous for distance to school d)discrete for # of subjects and qualitative for type of subjects e)discrete for
number of students f)discrete for cost and continuous for distance to school 3.a)continuous b)secondary 4.advantage: take less time than
collecting it yourself; disadvantage: unsure whether the sample is biased/representative/fair 5.it may not address the aims

Page 5 of 17
 Lesson: Types of Sampling Techniques:

_________________________________________________________________________
Every member of the population has an equal chance of being selected
EX: Lottery, draw names from an envelope/box

_________________________________________________________________________
There is an open invitation to any member of the population to participate in a survey or experiment

_________________________________________________________________________
Surveying members of the population that are easily accessible

_________________________________________________________________________
Selecting a sub-group as being representative of the entire population and every member of the sub-
group participates

_________________________________________________________________________
requires that the samples that are taken are destroyed in the process of testing
Ex. Cars used in crash tests

____________________________________________
Selecting members of the population randomly in stages
All
Ontario
Househol
ds

York
Toronto
Region

Toronto East Whitchur

Scarboro North Richmon Newmark
& East Etobicoke Markham Aurora King Vaughan Gwillimb Georgina ch
ugh York d Hill et
York ury Stouffville

____________________________________________
Selecting members that are in sequence at regular intervals
𝒑𝒐𝒑𝒖𝒍𝒂𝒕𝒊𝒐𝒏 𝒔𝒊𝒛𝒆
𝒊𝒏𝒕𝒆𝒓𝒗𝒂𝒍 =
𝒔𝒂𝒎𝒑𝒍𝒆 𝒔𝒊𝒛𝒆

Cities Number of drivers Sample size

_________________________________________________________ Markham 4088

Selecting members randomly from a population that Richmond Hill 5409

may be divided into groups (strata) that share a North York 2459
common characteristic (such as gender, grade,
education) such that the proportional representation of Scarborough 3236
a strata in the sample is the same as the proportional Total 15 192
representation in the population

Ex: survey 25% of the drivers in nearby 4 cities

Page 6 of 17
Practice for Sampling Techniques:
Identify the population and sampling techniques for each of the following scenarios.

Example1. For each of the following scenarios, identify the population and the type of sampling used.

a. A grade 12 class wants to determine the average height of the students in the class. The names of all
students in the class are placed in a hat and then names are drawn.

b. A Canadian publisher wants to know which type of book high school students like best. The publisher
chooses ten high schools at random and then surveys every student in each school.

c. You want to collect data about which TV shows teenagers like best. You ask your friends and teenage
cousins which shows are their favourites.

d. The student council wants to know how much money students are willing to pay for a yearbook. They
choose five boys and five girls from each period 1 class, from each grade, and survey them.

e. Police want to know the speed motorists drive along Bayview Avenue. They set up radar and
measure the speed of every 10th vehicle for a day.

Example 2. Before booking bands for the school dances, the Student Council at Bayview Secondary School
wants to survey the music preferences of the entire student body. The following table shows the
enrollment at the school.

Grade Number of Students Sample

9 455

10 432

11 409

12 384

Total
a. Design a stratified sample for 20% of the student body

b. You have been given an alpha list of students and due to time-constraint, a sample size of about 50
students is deemed to be sufficient. Suggest a method for selecting a systematic sample.

Page 7 of 17
 Sampling Techniques Homework Do #1-4, 7-9 7. Communication A community centre chooses 15 of its
Practice A members at random and ask them to have each
1. Identify the population for each of the following member of their families complete a short
questions questionnarire.
a. Who should be the next president of the a. What type of sample is the community centre
students’ council? using?
b. Who should be next year’s grade-10 b. Are the 15 community-centre members a
representative on the student council? random sample of the community? Explain.
c. What is your faviourite soft drink? c. To what extent are the family members
d. What Beatles song was the best? randomly chosen?
e. How effective is a new headache remedy? 8. Application A students’ council is conducting a poll of
2. Choose the sampling method used in each of the students as they enter the cafeteria.
following scenarios. a. What sampling method is the student council
a. A radio-show host invites listeners to call in using?
with their views on banning smoking in b. Discuss whether this method is appropriate
restaurants. for surveying students’ opinions on
b. The Heritage Ministry selects a sample of i. The new mural in the cafeteria
recent immigrants such that the proportion ii. The location for the graduation prom
from each country of origin is the same as for c. Would another sampling technique be better
all immigrants last year. for either of the surveys in part b>?
c. A reporter stops people on a downtown street 9. Application The host of a call-in program invites
to ask what they think of the city’s lakefront. listeners to comment on a recent trade by the Toronto
d. A school guidance counselor arranges Maple Leafs. One caller criticizes the host, stating that
interviews with every fifth student on the the sampling technique is not random. The host
alphabetized attendance roster. replies: “So what? It doesn’t matter!”
e. A statistician conducting a survey randomly a. What sampling technique is the call-in show
selects 20 cities from across Canada, then 5 using?
neighbourhoods from each of the cities, and b. Is the caller’s statement correct? Explain.
then 3 households from each of the c. Is the host’s response mathematically correct?
neighbourhoods. Why or why not?
f. The province randomly chooses 25 public Answers Section 2.3 pp. 117-118
Practice A
schools to participate in a new fundraising
1. a) students in the particular school
initiative. b) grade 10 students in the particular school
3. What type(s) of sample would be appropriate for c) you
a. A survey of engineers, technicians, and d) all those who have listened to the Beatles’ music
managers employed by a company? all those who have tried the new remedy
b. Determining the most popular pizza topping? 2. a) voluntary-response sample b)stratified sample
c) convenience sample d) systematic sample
c. Measuring customer satisfaction for a e) multi-stage sample f) cluster sample
department store? 3. a) stratified sample
Apply, Solve, Communicate B b) simple random sample, convenience sample
4. Natasha is organizing the annual family picnic and c) voluntary-response sample
wants to arrange a menu that will appeal to children, Apply, Solve, Communicate B
!"
teens, and adults. She estimates that she has enough 4. sample ≐ 32% of the members of each group; 4 children, 3
#$
time to survey about a dozen people. How should teens, 5 adults
%
Natasha design a stratified sample if she expects 13 5. Obtain a list of the students in the school. Calculate names.
"&
%
children, 8 teens, and 16 adults to attend the picnic? Then, select every 𝑡ℎ name.
"&
5. Communication Find out, or estimate, how many 6. Answers may vary.
students attend your school. Describe how you would 7. a) cluster sample
b) No, not every member of the community had an equal chance
design a systematic sample of these students. Assume of being surveyed. To the same extent as the member of the
that you can survey about 20 students. community centre was randomly selected.
6. The newly elected Chancellor of the Galactic 8. a) convenience sample
Federation is interested in the opinions of all citizens b) i) yes ii) no
regarding economic conditions in the galaxy. c) In part ii), a simple random sample should be used.
9. a) voluntary-response sample
Unfortunately, she does not have the resources to visit b) yes
every populated planet or to send delegates to them. c) not if the host claims that the callers are representative of the
Describe how the Chancellor might organize a multi- population
stage sample to carry out her survey. 10. Answers will vary.
11. Answers will vary.
12. Answers will vary.

Page 8 of 17
Lesson: Data Collection Bias
Collection Bias: Statistical bias is any factor that favors certain outcomes or responses and hence
systematically influences (or skews) the results of a survey.

_______________________________________________________________
Sampling bias occurs when the sample does not reflect the characteristics of the population. This can
result from poor sampling technique or data collection methods. This should be reduced by planning the
study utilizing the appropriate sampling technique.

______________________________________________________________Household bias occurs when one type of Household

Bias respondent is over represented because different sized groupings are polled equally.

______________________________________________________________
This is a form of sampling bias caused when a particular group or groups are underrepresented in the
survey because they have chosen not to participate.

______________________________________________________________
If the respondent deliberately gives false or misleading answers to the survey this would Response Bias
be called response bias. The dishonesty in the answers may be caused by embarrassment or fear to
answer the question honestly.

______________________________________________________________
Measurement bias is a consistent measurement error which skews the results of the survey. Often the
data collection process affects the variable that is being measured. Someimes these effects (if known) can
be accounted for in the analysis.
Two types of measurement bias questions:

(a) ___________________________________________________
is a question that suggests an answer which otherwise may not have been chosen by the respondent
without prompting.

(b) ____________________________________________________
is a question contains wording or information which influences the respondents.

Page 9 of 17
Practice for Data Collection Bias:
Choose the MOST obvious type of bias.

a) A survey asked students at a high school basketball game whether a fund for extracurricular activities
should be used to buy new equipment for the basketball team or instruments for the school band.

Types of Bias Explanation Solution

b) A social science class asks every tenth student entering the cafeteria to answer a survey on
environment issues. Less than half complete the questionnaire. The completed questionnaires show that
a high proportion of the respondents are concerned about the environment and about environmental
issues.

Types of Bias Explanation Solution

c) A civil engineer suggests that a cost-effective way to survey traffic speeds on a highway would be to
have a polic officer patrol the highway and record the speed of the traffic every half hour.

Types of Bias Explanation Solution

d) An aid agency in a developing country wants to know what proportion of households have at least
one personal computer. One of the agency’s staff members conduct the survey by calling household
randomly selected from the telephone directory.

Types of Bias Explanation

Solution
e) The following questionnaire was developed by Sarah’s friends”
Election Survey Sponsored by the friends for the election of Sarah committee.
Circle the appropriate response.
Gender: Male Female

Grade: 9 10 11 12 On election day, I intend to vote for:

Sarah Zahir Melanie

I want more:
dances hat days holidays fun
Types of Bias Explanation Solution

f) One student from each home room is surveyed about which new menu choices they would prefer in
the cafeteria. Below is information about the number of people in each home room: 8 Home rooms with
1-12 students 42 Home rooms with 13-24 students
20 Home rooms with 25-36 students

Types of Bias Explanation Solution

Page 10 of 17
 g) A teacher finishes explaining a new concept to the class and wants to check that all the students have
grasped the concept. The teacher asks those who do not understand to raise their hands.
Types of Bias Explanation
Solution
h) As part of a survey of the “Greatest Hits of All Time,” a radio station asks its listeners: Which was the
best Elvis songs? Blue Suede Shoes; Hound Dog; Heartbreak Hotel; All Shook Up; Suspicious Minds

Types of Bias Explanation Solution

i) Members of a golf and country club are polled regarding the construction of a highway interchange on
part of their golf course.
Types of Bias Explanation
Solution
j) A group of candidates running for the position of Prime Minister are asked whether they have ever
taken part in an illegal protest.
Types of Bias Explanation Solution
k) A random poll asks the following question: “The proposed nuclear power plant will produce a
number of jobs and economic activity in and around your community. Are you in favour of this forward-
thinking initiative?”

Types of Bias Explanation Solution

l) A survey uses a cluster sample of Toronto residents to determine public opinion on whether the
provincial government should increase funding on public transit.
Types of Bias Explanation Solution
m) A researcher appears on a talk-show and conducts an on-air survey about re-instituting capital
punishment in Canada.

Types of Bias Explanation Solution

n) Parents of high school students were asked: “Do you think that students should be released from
school an hour early on Friday, free to run around and get into trouble?”
Types of Bias Explanation
Solution
o) Audience members at an investment workshop are asked to raise their hands if they have been late
with a bill payment within the last six months.

Types of Bias Explanation Solution

Answers:
a) Sampling
b) Non-response i) Sampling
c) Response j) Response
d) Non-response k) Measurement
e) Measurement l) Sampling
m) Non-response
f) Sampling
n) Measurement
g) Response
h) Measurement o) Response

Page 11 of 17
Data Collecting Bias Homework Do #1-3,5,6,7
Practice A ii. Answer the question that follows your
1. Classify the bias in each of the following scenarios. mark in data management.
a. Members of a golf and country club are polled Over 80%: How many hours per week
regarding the construction of a highway do you spend on the Internet at home?
interchange on part of their golf course. 60-80%: Would home Internet access
b. A group of city councilors are asked whether improve your mark in data
they have ever taken part in an illegal protest. management?
c. A random poll asks the following question:
Below 60%: Would increased Internet
“The proposed casino will produce a number
of jobs and economic activity in and around access at school improve your mark in
your city, and it will also generate revenue for data management?
the provincial government. Are you in favour c. Suppose the goal is to convince the school
of this forward-thinking initiative?” board that every data management student
d. A survey uses a cluster sample of Toronto needs daily access to computers and the
residents to determine public opinion on Internet in the classroom. How might you
whether the provincial government should alter your sampling technique to help achieve
increase funding for the public transit. the desired results in this survey? Would
Apply, Solve, Communicate these results still be statistically valid?
2. For each scenario in question 1, suggest how the 6. Application A talk-show host conducts an on-air
survey process could be changed to eliminate bias. survey about re-instituting capital punishment in
3. Communication Reword each of the following Canada. Six out of ten callers voice their support
questions to eliminate the measurement bias. for capital punishment. The next day, the host
a. In light of the current government’s weak claims that 60% of Canadians are in favour of
policies, do you think that it is time for a capital punishment. Is this claim statistically valid?
refreshing change at the next federal Explain your reasoning.
election? 7. a) Locate an article from a newspaper, periodical,
b. Do you plan to support the current or Internet site that involves a study that contains
government at the next federal election, in bias.
order that they can continue to implement b) Briefly describe the study and its findings.
their effective policies? c) Describe the nature of the bias inherent in the
c. Is first-year calculus as brutal as they say? study.
d. Which of the following is your favourite male d) How has this bias affected the results of the
movie star? study?
i. Al Pacino e) Suggest how the study could have eliminated
ii. Keanu Reeves the bias.
iii. Robert DeNiro 8. Inquiry/ Problem Solving Do you think that the
iv. Jack Nicholson members of Parliament are a representative
v. Antonio Banderas sample of the population? Why or why not?
vi. Other:
4. Communication Answers Section 2.4, pp. 123-124
Practise
a. Write your own example of a leading 1. a) sampling bias b) response bias
question and a loaded question. c) measurement biase – loaded question
b. Write an unbiased version for each of these d) sampling bias-the cluster may not be representative
two questions. 2. a) A random sample of area residents should be taken.
b) The responses could be returned anonymously.
5. A school principal wants to survey data- c) Remove the preamble to the question and the phrase “forward-
management students to determine whether thinking.”
having computer Internet access at home d) A random sample of area residents should be taken.
improves their success in this course. 3. Answers may vary.
a) Which party will you vote for in the next federal election?
a. What type of sample would you suggest? b) same as a)
Why? Describe a technique for choosing the c) Do you think first-year calculus is an easy or difficult course?
sample. d) Who is your favourite male movie star?
b. The following questions were drafted for the e) Do you think that fighting should be eliminated from professional
hockey?
survey questionnaire. Identify any bias in the 4. Answers may vary.
questions and suggest a rewording to 6. Not statistically valid; the sample may contain response bias and also
eliminate the bias. sampling bias. The radio station may have a conservative audience.
i. Can your family afford high-speed 7. Answers may vary. 8. Answer may vary.
Internet access?

Page 12 of 17
Lesson: Creating a Survey
Format of a Survey/Questionnaire
(a) _________________________________________________________________
(b) _________________________________________________________________

(c) Survey Design:

Sample Survey Format
• Keep It Short and Simple
• Types of questions (* definitions look up on your own)
Closed-ended questions:
o Factual
o Rating scale
o Likert-type
o Semantic differential
o Multiple-choice/Dichotomous
o Rank order

Open-ended questions:
o Opinion and attitudes – questions related to
opinions, values, and attitudes that are
subjective about the thought processes and
feelings of a respondent

(d) _________________________________________________________________

Characteristics of a Good Survey

• Includes questions on all subjects essential to the research problem
• Does not include questions not related to the research problem
• Appears brief and easy to complete
• Respondents should feel like they are participating in an important project
• No questions that could bias the answers
• Designed to elicit clear and precise answers to all questions
• Phrasing, structure, and layout designed with the problem of tabulating the findings in mind

Characteristics of a Poor Survey

• Range of responses for questions is too narrow
• Misunderstood by respondents
• Too vague
• Ask information the respondent cannot remember or does not have
• Wrong level of intimacy
• Takes too much for granted
• Double negatives
• Overlong lists
• Overlapping alternatives

Page 13 of 17
Survey question types and survey structure
The types of survey questions used in a survey will play a role in producing unbiased or relevant survey responses.
As the survey designer, consider the types of questions to use and when it is appropriate to use them. Question
types range from open-ended (comments to essays) to closed-ended (yes/no, multiple choice, rating scale, etc). In
the end, it is the question types that determine what type of information is collected.

Question Types If you have a question in which you survey. This flow will also ease or
need the respondents to indicate what arouse the respondent’s interest and
1. Open-Ended Types items are the “most important” to “least overcome his/her doubts about the
important” then you can set up a survey’s intent. As a general guideline,
ranking question. there are three areas regarding
Open-ended questions are those that question sequence: opening questions,
allow respondents to answer in their question flow, and location of sensitive
own words. Open-ended questions 4. Rating Types
questions.
seek a free response and aim to
determine what is at the tip of the The rating type questions are used
respondent’s mind. These are good to when surveying the frequency of 1. Opening questions – The first few
use when asking for attitude or something like behavior or attitude. It is questions in the survey should be easy
feelings, likes and dislikes, memory best to present the rating scale in a and interesting in order to calm any
recall, opinions, or additional logical or consistent order. Therefore, it participants’ suspicions about the
comments. However, there can be makes sense to order the ranking or survey’s integrity. This allows the
some drawbacks to using open-ended rating choices from low to high (e.g. participants to build up confidence in
questions: Strongly Disagree to Strongly Agree the survey’s objective. In return, this
going from left to right). may stimulate their interest and overall
participation.
• Sometimes respondents may
find it difficult to express their If you set up the rating scale in your
survey in this format of “Strongly 2. Question flow – The question
feelings. This can result with
Disagree” to “Strongly Agree” make sequence in the survey body should
respondents answering “I
sure that the rest of the survey is take on a flow of ideas and be geared
don‟t know” or skipping it.
towards the respondents’ abilities.
• They do take more time and consistent and all rating scales go from
After you have established the first
effort to fill out and at times the low to the high frequency
throughout (or vice versa). In addition, general topic, all related questions
they can have a larger skip
some surveys may only label the should come up before a second topic
rate.
outliers or endpoints of the scale, but it is raised. It is a good idea to use
• In addition, analyzing open- “pages” in the online design to house
ended comments can be time is good practice to
each different section of the survey.
consuming and difficult. 2. Here you can raise one topic on one
Closed–Ended Types A five-point rating scale typically gives
page and include the
(Multiple Choice – One sufficient discrimination and is easily
instructions/information for this section
Answer or Multiple Answers) understood by survey participants. in the Page Description area. When
This is usually recommended for most
you are then ready to introduce a new
2. Closed-ended questions are those survey settings. However, there is no
topic to the survey, you can create a
with pre-designed answers with a set limit on the number of categories to new or second page to include that
small or large set of potential choices. use. Using too few could give less- page’s description and purpose.
One type of closed-ended question is a cultivated information, while using too
Conditional or Skip Logic questions are
“dichotomous” question which allows many could make the question hard to
also a good way to control the
respondents to choose one of two read and answer. Related to this setup respondent’s flow or route through the
answer choices (e.g. Yes or No), while is the decision of incorporating a survey. You can apply question or
another type is the “multi-chotomous” “middle category”. page skip logic to the survey when you
question, which allows respondents to want to guide respondents and
choose one of many answer choices. Question Sequence exclude them from certain pages of
questions that do not apply to them.
3. Ranked or Ordinal Questions A good survey design should help to
stimulate recall (if necessary); it should 3. Location of sensitive questions –
Ranking questions are best to use motivate the respondent to reply; and Some suggest that sensitive questions
when all the choices listed should be the survey should flow in an orderly should not be included at the
ranked according to a level of fashion. The sequence of questions beginning of the survey. However,
specification (e.g. level of importance). will help to create a certain flow to the there are no set rules on this. If you do

Page 14 of 17
include sensitive questions at the know how the survey works. From help to build rapport with the
beginning of the survey, then you may here respondents will not have to look respondent; possibly increasing the
run into respondents rejecting the back and forth in the survey to see likelihood that they will participate in
survey and exiting early. They may not what they are supposed to do. your future survey invites.
have built up confidence yet in the
survey’s integrity quite so early. b. Body of the Survey Design 2. Layout for coding and identification
Questions like demographics or
personal information are usually best
to introduce towards the end of the The use of space throughout the As the designer of the survey, pay
survey. This way, respondents are survey is also important. Trying to fit attention to the physical layout of the
likely to have already developed too much information (e.g. too many survey to reduce the likelihood of
confidence in the survey’s objective. questions) on a single page may cause errors on the respondents‟ end or on
respondents to struggle through the your end regarding areas of coding or
survey. If your survey has multiple editing. Here are some principles to
sections or parts, then it is good to follow to make the survey logical for all
introduce each new section as people accessing the survey, as well
Survey Layout suggested previously. Keep in mind to as easy to identify, code, and store:
make the sections and questions flow
1. Basic guidelines in a sequential order that makes sense 1. Identification – You can add a
to the respondents. unique number or identifier to each
a. Introduction questionnaire.
Here are some tips to remember when
designing the look of your survey: 2. Numbering – Questions can be
When designing your survey structure,
the overall format and layout is numbered sequentially throughout the
important from beginning to end. A 1. Make the survey visually appealing survey, even if the survey is divided by
poorly organized survey may cause and user-friendly. pages or sections. You can choose to
respondents to skip questions or 2. Try not to use small fonts or fonts have our tool number the questions
completely opt out of answering your that are not easy to read. Some throughout the entire survey as a
survey. It is good practice to begin participants may have a difficult time whole or have the questions numbered
your survey with an introduction that reading small print. according to each individual page. This
explains the survey’s purpose. Within 3. To avoid clutter, use white space. may help you in coding your survey.
the introduction, you may want to 4. Ask only one question per line.
include the name of the organization 5. Group similar question together or in 3. Instructions – General instructions
conducting the survey, the the same area of the survey. are important for the administration of
confidentiality information, and how the 6. Ask interesting questions in the the survey as well as for the collection
data collected will be used. Many beginning of the survey to grab the of accurate data. The following two
participants like some kind of participants‟ attention. This helps to types of information ought to be
assurance in regards to their stimulate interest. distinguishable in the survey:
responses; providing that kind of 7. Place demographic and/or sensitive questions to be read / answered and
information before the survey starts questions at the end of the survey. If instructions to be followed. You may
can help ease those concerns. You they are in the beginning, participants want to customize your survey to
may also want to provide an estimate may opt out early. include different fonts for the
of how long the survey might take or 8. Finally, test the survey before going instructions or page descriptions vs.
whether you are offering any kind of live. A small sample of test the survey questions themselves.
incentive or prize for taking the survey. respondents can help verify if your Place any special instructions on either
Remember to deliver on your promised survey is working properly. This the page description/section or directly
gift! If you provide this information up enables you to revise and edit above the question itself.
front it usually leads to honest questions and the survey design.
responses and more completed 4. Fonts & Formats – If you want to
surveys. c. End of Survey or Thank You Page emphasize important words, then
underline or bold them in the survey
Providing general instructions on how Once your respondent has reached the question or page description. This
to progress through the survey in the end of your survey, you can create a makes it easier for your respondents to
introduction or within each new section Thank You page. Here you can thank identify key points or items.
is important in letting your audience the respondent for their time. This may

Page 15 of 17
 Scales are more than a little important.
4 Common Sense Tips for
Creating Surveys that Work Rather than asking respondents a basic yes or no
question, use scales that measure both the direction
and the intensity of opinions.

Author: Scott Smith, Ph.D.|December 31, 2012 This is critical for research.

Someone who “Strongly Supports” a decision is very

Creating online surveys is as much an art as it is a different from someone who only “Slightly Supports” it.
science.
Scales extend the power of analysis from basic
It involves attention to detail in the design and flow of percentages to high-level analyses based on means and
your survey. Creating an effective survey that yields variance estimates (like t-test, ANOVA, regression, and
actionable insights can be difficult. the like).

Effective survey design and flow gives power to your Use scales whenever you can. You will get more
research. But great questions are the foundation for information from each question.
great research.

There are fundamental best practices for creating (3) Keep Coded Values Consistent
survey questions that all researchers must know.
Every survey response, option, question, or answer is
Here are 4 tips for creating surveys that work. coded as a numeric value that is reported as a percent
of responses or as a mean, median, range, etc.

(1) Keep It Simple These values are the basis for analysis.

Do you remember taking the SAT or ACT? It’s a long • Mean: Often referred to as an average, it is the
and boring process. sum of all the values divided by the number of
values.
Your average survey respondent can start to feel that • Median: The middle point in a data set. To
way about 15 minutes into a survey. Fifteen minutes is a determine the median, lay out a distribution
good upper-limit for most surveys. from lowest to highest and select the middle
value.
When a survey is too long, three bad things can happen: • Range: The highest and lowest data points in a
distribution form the range. VARIANCE: A
1. Respondents drop out: They simply quit taking dispersion measure of how far a set of numbers
the survey. It costs money to find respondents, is spread out.
and a high drop-out rate can not only cost a lot, • Example: Assuming we have data points 1, 2,
but can influence the quality of your results. and 6: Mean: 3 = (9 / 3) Median: 2 Range: 1-6
Having a reward for completion can reduce Variance: 7
drop-outs, but you can’t stop it completely.
2. People stop paying attention: Remember your Values must be coded consistently. Generally, we
elementary-school classmate who just filled in assign the highest value to the best outcome (ie
random bubbles during a test? He grew up. If it “Strongly Agree” that customer service is responsive)
takes too long to take your survey, he might do and then move down from there.
it again. We actually see this a lot, and
encourage researchers to use attention filters. For simplicity, keep your scale direction consistent
3. Clients get angry: The irony of upsetting throughout your survey. This makes it easier for
customers with an overly long satisfaction respondents to answer and for you as a researcher to
survey is not lost on your respondents. conduct your analysis.
The best way to collect quality data is to keep
your surveys short, simple, and well organized. If scales have the same scale of points, you can quickly
compare responses to different questions. For example,
if a survey asks respondents to rate a series of
(2) Use Scales Whenever Possible

Page 16 of 17
statements from Strongly Disagree to Strongly Agree, Here are some quick examples:
the responses are given these values:
1. Make it specific to them: With employee
evaluations, you can explain that feedback will
be used to determine awards, promotions, and
pay raises and will help management make
organizational decisions that will affect them.
Standard scaling helps managers to quickly understand 2. Explain unexpected questions: For instance, if
customer service ratings by simply looking at averages. it’s important for you to ask toy store customers
their preferred color of jeans, you might want to
For example, once managers understand that a 5-point explain why that is relevant.
agreement scale is being used, they could be given the 3. Justify requests for sensitive information: For
mean results for the following customer evaluation instance, you can explain that purchasing habits
(agreement) statements: will only be analyzed in aggregate for
benchmarking purposes or that results will not
• I am completely satisfied with the customer be shared outside your organization.
service — 3.15
• The customer service is prompt — 4.12 So what do you think? What are some tips you’d give to
• Customer service representatives are polite — fellow researchers? Leave a comment below and let’s
4.67 discuss.
• Customer service representatives are
knowledgeable — 2.08

Since all the statements are positive and the values are
scaled consistently, a higher mean reflects better results
in that area. A manager can look at these means and
quickly identify the 2.08.

We see that customer service representatives are

prompt and polite, but they don’t seem to know what
they’re talking about. As a result, overall satisfaction
with customer service is perhaps much lower than it
could be.

You can reverse scales (or word questions negatively) to

encourage respondents to read more carefully.

However, if you use reversed scales or negative wording

for some items, be sure to recode the scales so that all
scales point in the same direction. This will allow you to
quickly compare multiple areas of customer service.
(You can do recodes easily in Qualtrics.)

The simplest solution is just to keep all scales

consistent throughout every survey.

(4) Explain Why

Respondents are more likely to help you if they see

something of positive value for them.

Value offerings can range from a very general altruistic

appeal for their help to a very specific offer of an
economic incentive. For instance, with a customer
feedback survey, you can explain that feedback will help
improve customer service.

Page 17 of 17