SIA 2101

RESEARCH METHODS

Lecture 7: Methods of Data Collection

Dr. Phil Nyoni

 Objectives
• What information will you collect? (types of data,
links to questions/variables)
• What instrument(s) will you use? (selecting an
instrument, scales of measurement, validity,
reliability)
• Choosing a method of data collection
• Structure of the research
 Obtaining Permission

• Institutional or organizational (e.g. school

district)
• Site-specific (e.g. secondary school)
• Individual participants or parents
• Campus approval (e.g. university or college) and
Institutional Review Board (IRB)
 What data to collect? Types of data
measures
• An instrument is a tool for measuring,
observing, or documenting quantitative data
• Types of Instruments
– Performance Measures (e.g. test performance)
– Attitudinal Measures (measures feelings toward
educational topics)
– Behavioral Measures (observations of behavior)
– Factual Measures (documents, records)
 Linking data collection to variables
and questions

Flow of Activities Example

Identify the variable Self-efficacy for learning from
others
Level of confidence that an
Operationally define the variable
individual can learn something by
being taught by others

Locate data (measures,

observations, 13 items on a self-efficacy
documents with attitudinal scale from Bergin
questions and scales) (1989)

Collect data on Scores of each item ranged from

instruments yielding 0-10 with 10 being “completely
numeric scores confident.”
Locating or Developing an Instrument

• Look in published journal articles

• Run a Google Scholar search and use the
term “instruments” and the topic of the study
• Go to the web site for Evaluation and
Assessment
• Examine guides to commercially available
tests
• Develop your own instrument
 Criteria for choosing a good instrument:
Scales of Measurement
• Nominal (Categorical): categories that describe
traits or characteristics participants can check

• Ordinal (Categorical): Data is ranked according to

order of importance
- Likert scales, Semantic scales - Used to measure
behaviour, attitudes, opinions and preferences
 Criteria for choosing a good instrument:
Scales of Measurement
• Interval (Continuous): Can measure the interval
or distance between two points on a quantitative
instrument (eg thermometer)

• Ratio (Continuous): a scale with a true zero and

equal distances among units
• Has an absolute zero
• Is used when using weight, length and time
• Not used to measure people’s attitudes
 Measurement errors
1. The measuring instrument we use could bias
the actual data collected and interpretation of
findings
2. Measurement error
– An inaccuracy that influences the measurement of
an attribute
– Systematic error
- People tend to give a favorable impression
- This can bias their responses systematically
 Data collection

• Time needed for data collection

– Limit initial collection to one or two
observations or interviews
– Time is needed to establish a substantial
data base
• Obtaining permission to use materials
• Ethical issues
– Anonymity of participants
– Convey true purpose of study without
deception
 Using existing data

1. Do a literature survey (using sources of existing

data)

2. Research which is fully based on analysing

existing data

3. Advantages of using existing data

• Unobtrusive
• Practical – takes less time and costs less
• Reliability and validity can be confirmed
 Using existing data
1. Sources of existing data
• Primary data – info collected at the original source
• Secondary data – info taken someone other than the
original source eg another researcher’s report
• Tertiary data – interpretations based on comments on
secondary sources
2. Check and verify the authenticity and credibility of
sources and data
• Genuineness and credibility
 Documents

• Public and private records

• Good source for text data
• You must obtain permission before using
documents
• Optically scan documents when possible
 Audio-Visual materials

• Determine the material that can provide

evidence to address your research questions
• Determine if the material is available and
obtain permission to use it
• Check the accuracy and authenticity of the
material if you do not record it yourself
• Collect the data and organize it
Procedures for Administering the Data Collection

• Develop standard written procedures for

administering an instrument
• Train researchers to collect observational data
• Obtain permission to collect and use public
documents
• Respect individuals and sites during data
gathering (ethics)
 Quantitative Methods
• Experiment
• Questionnaires
 Experiments
• In an experiment, you test an idea (or practice
or procedure) to determine whether it influences
an outcome or dependent variable.
• A procedure carried out to support or refute a
hypothesis, or determine the efficacy or
likelihood of something previously untried.
• Assign individuals to experience it (and have
some individuals experience something different)
• Determine whether those who experienced the
activities (or practice or procedure) performed
better on some outcome than those who did not
experience it.
 Experiments
• Scientific experiment, testing a product,
evaluating a program, or exploring a
phenomenon.

• Examples
– Ad-targeting agency is conducting an experiment to
check what types of ads are to be shown to the user
to get the sales aka conversion

– Feature change on the website of some of e-

commerce site to assess which change users are
most receptive to
 Experiments Advantages
• Researchers have firm control over variables to obtain
results.
• The subject does not impact the effectiveness of
experimental research. Anyone can implement it for
research purposes.
• The results are specific.
• Post results analysis, research findings from the same
dataset can be repurposed for similar research ideas.
• Researchers can identify the cause and effect of the
hypothesis and further analyze this relationship to
determine in-depth ideas.
• Experimental research makes an ideal starting point. The
collected data could be used as a foundation to build new
research ideas for further studies.
 Types of Experiments
• Pre-experimental Research Design
• Includes one or more than one experimental groups
to be observed against certain treatments. It is the
simplest form of research design that follows the
basic steps in experiments. The pre-experimental
design does not have a comparison group.
 Types of Experiments
• True Experimental Research Design
• Investigates the cause-and-effect relationships
between the variables under study. The research
method requires manipulating an independent
variable, random assignment of participants to
different groups, and measuring the dependent
variable.
 Types of Experiments
• Quasi-experimental Research Design
• Aims to establish a cause-and-effect relationship
between an independent and dependent variable.
However, unlike a true experiment, a quasi-
experiment does not rely on random assignment.
Instead, subjects are assigned to groups based on
non-random criteria.
• This type of research design is used in field settings
where random assignment is either irrelevant or not
required.
Experiments
 A/B Testing
• Randomized experimentation is called AB testing which is
broadly categorized under the realm of hypothesis testing.

• A/B testing, also known as split testing, refers to a randomized

experimentation process wherein two or more versions of a
variable (web page, page element, etc.) are shown to different
segments of website visitors at the same time
A/B Testing
 Experimental procedures

• Used to establish cause and effect

– Between independent and dependent variables
– Control for all variables that might influence the
outcome
• Used when testing theories
• Used when comparing two or more groups
• Manipulation of the different conditions
 Pre-Test and Post-Tests

Intervention
Time 1 Time 2

Pre-Test Post-
Test
 Survey Method
Survey is a structured way of collecting
standardized information from individuals
using a questionnaire.
Basically 3 types of survey are used by
researches:
•Mail survey
•Telephone survey
•Internet (on-line) survey
 Use of a survey

• Assess trends
• Opinions, beliefs, and attitudes
• Follow-up analyses
• Evaluations
 Types of survey designs

Time of Data Collection

Study Over Time Study at One Point in Time

Longitudinal Cross-sectional

Changes
in a Changes
Trends in sub-population
in the Attitudes
the same group identified Community Program
same and
population by a common Needs Evaluation
people Practices
over time characteristic
over time
over time

Group National
Trend Cohort Panel
Comparisons Assessment
Characteristics of survey research
• Sampling from a population
• Collecting data through questionnaires or
interviews
• Designing instruments for data collection
• Obtaining a high response rate
 Mail Survey
• Most popular method
• Uses envelop and stamp, and self-addressed stamped
return envelop
• Enables you to choose a large sample
Advantages:
• Low cost
• Convenience
• No bias responses
Sampling–internal link
Disadvantages:
• Very low response rate
• Ability of respondent to answer survey
 Telephone Survey
Interviewer collects the relevant information from
the target respondents through telephone
conversation.

Helpful tips to locate the respondent:

•Repeat calls might be necessary if the
respondents work in organizations and the only
channel to reach them is through their secretaries
•As the researcher may not know the name and
designation of the respondents, there is every
possibility of interviewing the wrong person
 Telephone Survey (cont.)
• In a telephone survey, the interviewer collects the
relevant information from the target respondents
through telephone conversation.
• The researcher can intimate the respondents in
advance informing them about the telephone
interview on the subject matter.
 Telephone Survey (cont.)
Advantages:
• It is more flexible in comparison to mailing method
• It is faster than other methods of survey. It is a quick
way of obtaining the information
• It is relatively cheaper
• It can cover reasonably large numbers of people or
organisations with wide geographic coverage
 Telephone Survey (cont.)
• High-response rate—keep going till the required
number
• Recall is easy; call-backs are simple and
economical
• Interviewer can explain requirements more easily
• Replies can be recorded without causing
embarrassment to respondents
 Telephone Survey (cont.)
Disadvantages:
• This kind of survey is often connected with selling.
• The questionnaire must be short and questions
must be simple and straightforward, otherwise
respondents may refuse to answer them.
• Surveys are restricted to respondents who have
telephone facilities.
• Repeat calls are inevitable—average 2.5 calls to get
someone.
 Internet (on-line) Survey
The questionnaire can be distributed as mail
messages through attachment or posted as
World Wide Web forms in the internet or
distributed via publicly available computers in
high-traffic areas.
 Internet (on-line) Survey (cont.)
Advantages:
• Cost saving
• Ease of editing/analysis
• Faster transmission time
• Easy use of pre-letters
• Higher response rate
• More candid responses
• Potentially quicker response time with wider
magnitude of coverage
 Internet (on-line) Survey (cont.)

Disadvantages:
• Sample demographic limitations
• Lower levels of confidentiality
• Layout and presentation issues
• Additional orientation/instructions
• Response rate
 Comparison among different survey methods

Cost Mail survey Telephonic Internet Personal interview

survey survey
Cost Often lowest Moderate Moderate Usually highest
Ability to probe No personal No personal No personal Greatest opportunity for
contact or contact or contact or observation, building
observation observation observation rapport and additional
probing
Responden’s Yes Perhaps, but Yes Perhaps, if interview
ability to usually no time is prearranged with
complete at respondent
own
convenience
Interview bias No Some, No Greatest chance
perhaps due
to voice
inflection
 Comparison among different survey methods
Cost Mail survey Telephonic Internet Personal interview
survey survey
Ability to decide who Least Some No Greatest
actually responds to
the questions

Complex questions Least Somewhat Least More suitable

suitable suitable suitable

Visual aids Little No High Greatest opportunity

opportunity opportunity opportunity

Potential negative ‘Junk mail’ ‘Junk calls’ No Invasion of privacy

respondent reaction
 Questionnaires
A set of printed or written questions with a choice of answers,
devised for the purposes of a survey or statistical study.
Questionnaire is one of the popular tools for the quantitative
research.
Purpose is to get clear-cut, precise and accurate results,
factually reflecting the situation under study.

1.Questions which should be avoided:

• Double-barreled –two questions are joined in one question
– either / both, or/therefore/and
• Loaded language / leading questions – conveys a
position for or against a topic being investigated “also”
• “Can” or “might” should be avoided – only one logical
answer
• Negative items – could be misinterpreted
• Incomplete question – inadequately defined
 Questionnaire - Advantages

• Broad coverage (local, national, international)

• Frank and anonymous answers
• Economical for large population
• Less Time consuming
• Quantitative data (easy to analyze)
• Filling at a convenient time
• Easy to respond (fixed responses)
• Good for sensitive & ego-related Qs.
• Original hard-to-obtain data
 Questionnaire - Disadvantages
• No clarification for ambiguous questions
• Inadequate motivation to respond
• Poorly worded or direct questions
• Unattractive style and format
• Low response rate
• Inaccurate responses – no verification
• Limited fixed responses
• Cannot uncover causes or relationships for
attitudes, beliefs, actions
 Type of Questions

• Factual: Age, gender, education, experience (often

used to investigate relationship)
• Informative: What respondents know about a
given topic (How do you come to know about the
availability of e-journals?)
• Attitudinal/Opinion: To obtain info about
respondents’ beliefs, feelings, values (Do to agree
that current copyright law is fair?)
• Self-perception: Allows subjects to compare
their ideas or actions with others (How active are
you in the community work? How will you describe your
computing skills?)
 Type of Questions
• Standard of Action: How respondents will act in a
situation (For which party you will vote in the next
election? Will you join the organ donation society? )
• Projective Questions: Allow respondents to
answer in an indirect manner by imposing their
feelings, attitudes or beliefs on others (Are most of
the Singaporeans quitters or stayers? Are students happy
with the grading scheme?)
 Structured/Unstructured Qs

• Unstructured Questions: Allow respondents to reply

freely without having to select one of several provided
responses (also called open-ended questions)
• Useful for exploratory studies in which various
dimensions and facets of a problem are
examined
• Usually difficult to analyze responses
– What steps are required to improve the quality of
National Service?
• Usually low response rate
 Structured Questions

• Dichotomous: yes/no, true/false, agree/disagree

• Multiple Choice: What sources do you use for writing term
reports? (check all that apply)
– How many hours do you exercise per week (check only one
response)
– How satisfied are you with the quality of canteen ‘A’ food?
• Contingency: Determine if the respondent is qualified to
answer a subsequent question)
– Do you use databases available through iGems? (If no, please
move to question 15) *
– Have you participated in DIS orientation? (Yes/No)
If yes, how effective was this briefing? (Very effective, effective,
ineffective …)
 Scaling Responses
• Likert-type Scale: Each response is assigned a
numeric ranking based on a continuum that
contains predetermined units of measurement
• Designed to show a differentiation among
respondents opinions
– How important is …. (very important, important ….)
– How adequate are …. (v. adequate ……..inadequate)
– How frequently do you ... (frequently … infrequently)
 Scaling Responses
• Semantic Differential Scale: Provides a set of
bipolar adjective pairs
Q. How would you rate performance of the
Income Tax staff?
1 2 3 4 5 6 7
Bad … … … … … … … Good
Unfair … … … … … … … Fair
Harsh … … … … … … … Gentle
 Ordering of questionnaire items

• Begin with general or broader questions

• Items should follow a logical pattern
• Begin with neutral questions, put sensitive
questions more towards the middle
 Observations
• An Observation is the process of gathering first-
hand information by observing people and
places at a research site.
• Observational roles
– Participant observer
– Non-Participant observer
• Observational roles can be changed
 Observations

The researcher should keep certain things in

mind:
• What should be observed?
• How should the observations be
recorded? Or how should the accuracy of
observations be ensured?
 Types of Observation

• Observation is characterised by careful definition of

units to be observed, the style of recording the
observed information, standardised conditions of
observation and the selection of pertinent data of
observation. This is known as structured
observation.
• On the other hand, if the observation is to take place
without these characteristics being sought in
advance, it is known as unstructured observation.
 Observations
• Conduct multiple observations
• Record both descriptive and reflective field notes
during the observation
– Descriptive field notes describe the events, activities
and people
– Reflective field notes record personal reflections that
relate to their insights, hunches or broad themes that
emerge
 Observational protocols

• The header: essential information about the

interview
• Left column to record descriptive notes
• Right column to record reflective notes
• A picture of the site may be sketched
 Qualitative Methods
• Interviews
• Focus Group Discussions
 Interview
A qualitative research technique that involves conducting
intensive individual interviews with a small number of
respondents to explore their perspectives on a particular idea,
programme or situation.

Advantages:
Because interviews are flexible, it can provide researcher with
detailed information
Respondents do not have to be literate
Non verbal communication can be observed
Advantages telephone interview
Data can be collected quickly
Not expensive
 Interview
Limitations and pitfalls:
• Biased due to their stake in the programme
• Interviews can be a time-intensive evaluation activity
• Must be appropriately trained
• Generalisations about the results usually could not
be made
 Interviews
1. Structured - specific lists of questions are
asked
2. Unstructured interviews – there are no specific
questions, respondents give their reactions to
general issues
3. Two types of face to face interviews
– In-depth interviews – to obtain detailed information
– Focus group interviews (group interviews) –
consist of 6 to 12 people who are interviewed
together at the same time.
 Interviews
4. Types of questions asked in interviews
– Closed-ended questions – fixed number of answers
(yes/no)
– Open questions – encourage respondents to
express their attitudes
• Free-narration questions – tell his/her own story
• Role-playing questions – respondent is invited to
play a certain role
– Follow-up questions – follow on from the
respondents answers to a closed ended question
 Interviews
5. Conducting an interview
– Decide who is going to be interviewed
– Individuals or groups, social setting, time chosen
– Advanced planning
– Purpose of the interview must be made clear at
beginning of interview
– Begin with general questions, later sensitive ones
– Do a pre-test or pilot test before the actual
research
 Interviews
• Types: Personal, Phone, e-mail, Focus Group
• General open-ended questions are asked
– allows the participant to create options for responding
– participants can voice their experiences and
perspectives
• Information is recorded then transcribed for
analysis
Structured, unstructured, and semi-
structured interviews
Type of
Approach to
Response Types of Leading to
Data
Options to Interviews Data
Collection
Questions

Quantitative Closed- Structured/ Scores to

Ended semi-structured answers
Interviews

Qualitative Open- Unstructured Transcription

Ended Interviews of words
 Interview protocols
• The header: essential information about the
interview
• Open-ended questions include
– “ice-breaker”
– ones that address major research questions
– probes that clarify and elaborate
• Closing comments thanking the participant
 Focus Group Discussion

A carefully planned and moderated discussion

among individuals having common characteristics
to obtain the meaningful information on the area
of interest.

A focus group is a small group discussion guided

by a trained leader, used to learn more about
opinions on a designated topic, and then guide
future action.
How are focus groups different from
regular “groups”?

• They are focused on a specific topic

• They have a trained facilitator
• Members of the group are encouraged to talk
openly about their opinions and respond to other
members
 Case study

• Definition: an in-depth exploration of a

bounded system (time, place, physical
boundaries)
• Subject for case studies
– individual or several individuals
– series of steps that form a sequence of activities
• Researcher develops understanding of the
case by collecting multiple forms of data
• Researcher locates the “case” or “cases” within
their larger context
 Ethnographic research
• Ethnographic designs are qualitative research
procedures for describing, analyzing, and
interpreting a culture-sharing group’s shared
patterns of behavior, beliefs, and language that
develop over time.
 When do you conduct an
ethnography?
• Study of a group helps you understand a larger
issue
• Have a culture-sharing group to study
• Want a day-to-day picture
• Long-term access to culture-sharing group
 Types of Methods
• Qualitative • Quantitative
– Case Studies – Surveys
– Focus Groups – Experiments
– Interviews – Questionnaires
– Observations
– Ethnography
 Methods Approach
-adapted from Creswell

Tend to Qualitative Quantitative Mixed

Use these Case study, Surveys and Aspects of each
strategies narrative, experiments
interview,
ethnography
Employ these Open-ended Close-ended questions, Both open- and
methods questions, predetermined close-ended,
Text or image data approaches, numeric emerging and
data predetermined
approaches
Use these Positions himself Tests or verifies Collects both
practices Collect participant theories quantitative and
meaning Identifies variables qualitative data
Focus on single Relates variables and Develops rationale
concept questions for mixing
Study context Observes and measures Integrates data at
Interpret numerically different stages
Collaborate with Emphasis on unbiased Employs mixed
participants approach practices
Employs statistics
Criteria for choosing a good instrument:
Reliability
• Reliability: Scores from measuring variables
that are stable and consistent
• Example: Bathroom scale
• Types of reliability
– Test-retest (scores are stable over time)
– Alternate forms (equivalence of two instruments)
– Alternate forms and test-retest
– Inter-rater reliability (similarity in observation of a
behavior by two or more researchers)
– internal consistency (consistent scores across the
instrument)
Criteria for choosing a good instrument:
Validity

• Validity: Scores from measuring variables that

are meaningful
• Types of validity
– Content (representative of all possible questions
that could be asked)
– Criterion-referenced (scores are a predictor of an
outcome or criterion they are expected to
predict)
– Construct (determination of the significance,
meaning, purpose and use of the constructs)
 Validity and reliability
1. Validity:
A methodological requirement for
– research methods such as observation, the use of
measuring instruments
– research results such as the analysis and
interpretation of collected data and conclusions from
analysis
2. Reliability:
When identical investigations are repeated,
similar research results will be
obtained.
 Threats to internal validity

History
•Uncontrolled events that influence the outcome of
research
•An event unrelated to the study occurs during the study
and influences the dependent variable
•Ex during a study to evaluate people’s attitudes toward
space travel, a space craft explodes on the launch pad,
killing the astronauts
 Threats to internal validity

Instrumentation
• The instrument (questionnaire) used to measure
the dependent variable may change during the
study. The instrument may deteriorate, or
improve over time
• Ex if the springs of a scale weaken during a
weight-loss experiment, this will influence the
readings on the scale
 Threats to internal validity

Selection
• It can bias the study if there are important,
unsuspected differences between the subjects in
the two groups used for comparison.
• Ex an experiment on physical aggression in which
the one group unintentionally contains soccer and
rugby players, whereas the other group is made
up of musicians and painters
 External validity
1. The extent to which results can be generalised to
other populations or circumstances
2. Selection poses a threat to external validity
3. Subjects chosen for a study should be
representative of a larger population.
4. Reactive effect or Hawthorne effect: subjects
respond because they know they are being
observed
5. Researcher expectancy - expectations cause a
researcher to behave in a manner that makes the
expected event more likely to occur.
 Credibility & Trustworthiness
Making the study more believable

•Are the data based on one’s own observation, or is it

hearsay?
•Is there corroboration by other’s of the observation?
•In what circumstances was an observation made or
reported?
•How reliable are those providing the data?
•What motivations might have influenced a
participant’s report?
•What biases might have influenced how an
observation was made or reported?
 Triangulation

– Results of one method/research strategy can be cross-

checked against the results of another
– qualitative interviews conducted to ‘check and
correct’ the quantitative data
– Planned or unplanned
– What happens if results are inconsistent?
• treat one set of results as definitive
• but better to re-examine the data
THANK YOU