Day 3

Research Design &

Data Management

Qualitative Research
2© 2009

Introduction

• Data Collection Strategies

• Characteristics of Good Measures
• Quantitative and Qualitative Data
• Tools for Collecting Data
3© 2009

Data Collection Strategies

• No one best way: decision depends on:
– What you need to know: numbers or stories
– Where the data reside: environment, files, people
– Resources and time available
– Complexity of the data to be collected
– Frequency of data collection
– Intended forms of data analysis
4© 2009

Rules for Collecting Data

• Use multiple data collection methods

• Use available data, but need to know
– how the measures were defined
– how the data were collected and cleaned
– the extent of missing data
– how accuracy of the data was ensured
5© 2009

Rules for Collecting Data

• If must collect original data:
– be sensitive to burden on others
– pre-test, pre-test, pre-test
– establish procedures and follow them (protocol)
– maintain accurate records of definitions and
coding
– verify accuracy of coding, data input
6© 2009

Qualitative Approach
• Data that deal with description
• Data that can be observed or self-reported, but
not always precisely measured
• Less structured, easier to develop
• Can provide “rich data” — detailed and widely
applicable
• Is challenging to analyze
• Is labor intensive to collect
• Usually generates longer reports
7© 2009

How to Decide on Data Collection

Approach

• Choice depends on the situation

• Each technique is more appropriate in
some situations than others
• Caution: All techniques are subject to bias
8© 2009

Data Collection Tools

• Records and Secondary Data

• Observation
• Interviews
• Focus Groups
 Data collection in qualitative studies

• Qualitative researchers, typically rely on four

methods for gathering information:
– participation in the setting ,
– direct observation ,
– in depth interviews, and
– analysis of documents and materials
– focus groups and key informant interviews.

Qualitative Research 9
©
0 2009

Tool 1: Records and Secondary

Data
• Examples of sources:
– files/records
– computer data bases
– industry or government reports
– other reports or prior evaluations
– census data and household survey data
– electronic mailing lists and discussion groups
– documents (budgets, organizational charts, policies and
procedures, maps, monitoring reports)
– newspapers and television reports
©
1 2009

Using Existing Data Sets

Key issues: validity, reliability, accuracy,

response rates, data dictionaries, and
missing data rates
 Advantage/Challenge:
Available Data
Advantages Often less expensive and faster
than collecting the original data
again

Challenges There may be coding errors or

other problems. Data may not be
exactly what is needed. You may
have difficulty getting access. You
have to verify validity and
reliability of data
©
3 2009

Tool 2: Observation
• See what is happening
– traffic patterns
– land use patterns
– layout of city and rural areas
– quality of housing
– condition of roads
– conditions of buildings
– who goes to a health clinic
 Participant Observation
• Why do it?
– See what’s really going on
– Counterbalance: Triangulate data from other
methods
• Understand and “test” what people say
• Know what to ask people about because you’ve
seen it already

Roberta Goldman, PH.D., Brown University and HSPH, 2012

©
5 2009

Observation is Helpful when:

• need direct information
• trying to understand ongoing behavior
• there is physical evidence, products, or
outputs than can be observed
• need to provide alternative when other data
collection is infeasible or inappropriate
 Participant Observation
• How to really see:
– Know what you’re looking for, and
– Be open to seeing what you don’t expect
– Be cognizant of what you’re looking at
– Observe the details, variations, etc.
• Take notes
• Reflect on observations and notes; question what you
saw
• Discuss observations and notes
• Go look again
– and so on. . .
Roberta Goldman, PH.D., Brown University and HSPH, 2012
 Participant Observation
• Spectrum of observation

Full participant

Passive Observer

Roberta Goldman, PH.D., Brown University and HSPH, 2012

©
8 2009

Ways to Record Information from

Observations
• Observation guide
– printed form with space to record
• Recording sheet or checklist
– Yes/no options; tallies, rating scales
• Field notes
– least structured, recorded in narrative, descriptive
style
 Participant Observation - Fieldnotes
• Document your observations
– Fieldnote journal – running record of observations
AND observer comments
– Structured observation note grid
– Brief notations while in the field
– Expand upon and organize notes as soon as possible
– thick description
– Truism in anthropology:
• For every hour of observation you need 3 hours of writing
fieldnotes
Roberta Goldman, PH.D., Brown University and HSPH, 2012
 Steps in participant observation:
Strategies for taking field notes

• Include descriptions and interpretations of

individuals, interactions, and events
– Distinguish descriptions from interpretations
• Record time and location of observations, as well
as key information (weather, events happening
and their significance)
• Keep theoretical memos – which are the tentative
interpretations emerging and being assessed
through further data collection

20
 Field notes (continued)
• May not be possible or advisable to take notes
while in the field
– Important that they be done as soon after field
observation as possible
• Note-taking is time-consuming because it is
integral to guiding the data collection and
continuing the analysis
– e.g., field notes for When Prophecy Failed were well
over 1,000 typed pages

21
 Steps in participant observation:
Integrating data collection and analysis
• Organizing field notes into different types of files
facilitates data analysis
• Master field file – complete journal of field notes;
number pages and include entry dates
• Background, history file – subfile organizing
background material
• Key character files – subfiles on key players in the
group or organization
• Analytic files – subfiles for different types of
observations or relationships

22
 Advantages and Challenges:
Observation
Advantages Collects data on actual vs. self-
reported behavior or perceptions. It is
real-time vs. retrospective

Challenges Observer bias, potentially unreliable;

interpretation and coding challenges;
sampling can be a problem; can be
labor intensive; low response rates
©
4 2009

Tool 3: Interviews
• Excellent for asking people about:
– perceptions, opinions, ideas
• Less accurate for measuring behavior
• Sample should be representative of the whole
• Big problem with response rates
©
5 2009

Interviews

• Often semi-structured
• Used to explore complex issues in depth
• Forgiving of mistakes: unclear questions can
be clarified during the interview and
changed for subsequent interviews
• Can provide evaluators with an intuitive
sense of the situation
 In-depth interviews
• Some studies cannot employ the participant
observation method
• In-depth interviews allow participants to describe
their experiences and the meaning of events taking
place in their lives
– Verbatim quotes capture the language and meaning
expressed by participants
• Interviews are flexible and allow for probing
– Interview method is quite diverse, adaptive

26
 In-depth interviews (continued)
 Three key elements for the interview method to be successful:
1. Explicit purpose – researcher and informant are aware that the
discussion has a purpose
2. Ethnographic explanations – researcher tries out explanations on the
participants to see if they make sense
 Encourage the informants to use colloquial language, and teach the
researcher its meaning
3. Ethnographic questions include:
i. Descriptive questions – ask participants to describe their experiences (e.g.,
their ideas, circumstances, viewpoints, dilemmas, etc)
ii. Structural questions – ask participants how they organize their world (e.g.,
activities)
iii. Contrast questions – ask participants what is meant by specific terminology

27
 Interview do’s and don’ts
• Do listen more and talk less
• Do follow up on what is not clear and probe more deeply into what
is revealed
• Don’t use leading questions; do use open-ended questions
(“probes”)
• Don’t interrupt; do wait
• Do keep interviewee(s) focused
• Don’t be judgmental about or react to an interviewee’s opinions,
views, or beliefs
• Don’t engage in debate with an interviewee
• Do record everything the interviewee says and note impressions of
interviewee’s nonverbal behavior

28
©
9 2009

Challenges of Interviews
• Can be expensive, labor intensive, and time
consuming
• Selective hearing on the part of the
interviewer may miss information that does
not conform to pre-existing beliefs
• Cultural sensitivity: e.g., gender issues
©
0 2009

Tool 4: Focus Groups

• Type of qualitative research where small
homogenous groups of people are brought
together to informally discuss specific topics
under the guidance of a moderator
• Purpose: to identify issues and themes, not
just interesting information, and not “counts”
©
1 2009

Focus Groups Are Inappropriate

when:
• language barriers are insurmountable
• evaluator has little control over the situation
• trust cannot be established
• free expression cannot be ensured
• confidentiality cannot be assured
 Focus group interviews
• Interview format, but in a group setting
– 6-12 participants with common experience
• Dates back to the 1940s – used to assess
effectiveness of morale-boosting radio shows
– 1970s onward – used by market researchers
– 1980s onward – used by academics
• Transcript of discussion is the data
– Plus accompanying notes
– Use content analysis or grounded theory approach to
analyze the data

32
 Focus group interviews (continued)
• Strengths:
– Open-ended question
• Spontaneously deal with issues as they arise
– Cost-effective method of collecting data
– Less time-consuming
• Weaknesses:
– One or two participants may dominate
– Not done in a natural setting, so little “observation” to
help understand the experience of the participants

33
 Focus Group Process
Phase Action

1 Ice-breaker; explain purpose; ground rules;

Opening
introductions
2 Warm- Relate experience; stimulate group interaction;
up start with least threatening and simplest questions
3 Main Move to more threatening or sensitive and
body complex questions; elicit deep responses; connect
emergent data to complex, broad participation
4 Closure End with closure-type questions; summarize and
refine; present theories, etc; invite final comments
or insights; thank participants
 Advantages and Challenges of
Focus Groups
Advantages Can be conducted relatively quickly and
easily; may take less staff time than in-depth,
in-person interviews; allow flexibility to make
changes in process and questions; can
explore different perspectives; can be fun

Challenges Analysis is time consuming; participants not

be representative of population, possibly
biasing the data; group may be influenced by
moderator or dominant group members
©
6 2009

Data Collection Summary

Choose more than one data collection
technique
No “best” tool
Do not let the tool drive your work but rather
choose the right tool to address the
evaluation question
Qualitative Research