IPDET

Module 8:
Data Collection Methods
Participatory

Intervention Collect Available Data

Observation
or Evaluation Design Survey
Approaches
Policy Questions Questionnaire
Focus Groups
Diaries, Journals, Self-
reported Checklists
Expert Judgment
Delphi Technique
Citizen Report Cards
 Introduction
• Data Collection Strategies
• Data Collection General Rules
• Key Issues about Measures
• Quantitative and Qualitative Data
• Introduction to The Toolkit
• Techniques for Using Surveys
• Techniques for Using Focus Groups
IPDET 22
 No One Best Way
• Decision Depends On:
– What you need to know
• numbers or stories
– Where the data reside
• environment, files, people
– Resources and time available
– Complexity of the data to be collected
– Frequency of data collection
IPDET 33
 Structured Approach
• All data collected in the same way
• Important for multi-site and cluster
evaluations so you can compare
• Important when you need to make
comparisons with alternate interventions

IPDET 44
 When to Use Structured
Approach:
• You wish to be very precise
• You are working with a large sample or
populations
• You are sure of what you wish to measure
• You want to show your results numerically
• You need to make comparisons across
different sites or interventions

IPDET 55
 Semi-structured Approach
• Systematic and follow general
procedures but data are not collected in
exactly the same way every time
• More open and fluid
• Does not follow a rigid script
– may ask for more detail
– people can tell what they want in their own
way

IPDET 66
 When to Use Semi-
structured Approach:
• You are conducting exploratory work in a new
development area
• You are seeking understanding, themes,
and/or issues
• You want anecdotes, stories, or in-dept
information
• You are not sure of what you wish to measure
• There is no need to qualify

IPDET 77
 Data Collection
General Rules
• Use available data if they already exist
• If using available data, be sure to find
out how they:
– collected the data
– defined the variables
– ensured accuracy of the data

(continued on next slide)

IPDET 88
 Data Collection
General Rules (cont.)
• If you must collect original data:
– establish procedures and follow them
– maintain accurate records of definitions
and coding
– pre-test, pre-test, pre-test
– verify accuracy of coding, data input

IPDET 99
 Key Issues about
Measures
• Are your measures credible?
• Are your measures valid?
• Are you measuring what counts?
• Are your measures reliable?
• Are your measures precise?

IPDET 10
10
 Credibility

• How trustworthy or believable is your

data collection ?
– Are the data you are collecting giving you
information about the actual situation?
• Make sure data you are collecting are
relevant and most important
information

IPDET 11
11
 Validity
• Does the measurement actually measure
what it is supposed to?
• Two kinds of validity
– face validity
• contents of the test or procedure look like they are
measuring what they are supposed to measure
– content validity
• content of the test or procedure adequately represents all
that is required for validity

IPDET 12
12
 Reliability
• Term to describe the stability of your
measurement
• Measures the same thing, same way in
repeated tests
• Examples:
– in sports, speed measured by stopwatch
– birth weights of newborn infants
– attendance rates at schools
IPDET 13
13
 Precision
• How the language used in the data
collection matches the measure
• Example:
– if the question is about countries, the
measures must be at the national level
– if the question is about people, the
measures must be on the individual level

IPDET 14
14
 Quantitative Approach
• More structured
• Attempts to provide precise measures
• Reliable
• Harder to develop
• Easier to analyze

IPDET 15
15
 Qualitative Approach
• Less structured
• Easier to develop
• Can provide “rich data” — detailed and
widely applicable
• Is challenging to analyze
• Is labor intensive to collect
• Usually generates longer reports
IPDET 16
16
 Which Data?
If you: Then Use:
- want to do statistical analysis
- know exactly what you want to Quantitative
measure
- want to cover a large group
- want anecdotes or in-depth information
- are not sure what you want to measure Qualitative
- do not need to quantify

IPDET 17
17
 Obtrusive vs. Unobtrusive
Methods
• Obtrusive – observations are made of
behavior with the participant’s
knowledge
– interviews, surveys, focus groups
• Unobtrusive – observations done without
the knowledge of the participant
– historical/document/archival data
– watching participants at a distance

IPDET 18
18
 How to Decide on Data
Collection Approach
• Choice depends on the situation
• Each technique is more appropriate in
some situations than others
• Caution: All techniques are subject to
bias

IPDET 19
19
 Combinations
• Can use a variety of data collection
approaches to answer different
questions or for multiple sources
• Triangulation: collect same information
using different approaches

IPDET 20
20
 Data Collection
• Participatory Data Collection
• Available Data
• Observation
• Surveys
• Focus Groups
• Diaries, Journals, Self-reported Checklists
• Expert Judgment
• Delphi Technique
• Citizen Report Cards
IPDET 21
21
 Toolkit 1: Participatory
Data Collection
• Data that are collected when interacting
with people
• Examples:
– transect walks
– social mapping
– community meetings

IPDET 22
22
 Mapping
• Collect and plot information on the
distribution, access, and use of
resources within a community
• Useful tool to involve stakeholders
– provides a way to work together
– increases understanding of the community
– generates discussions, verifies secondary
sources of information, perceived changes
IPDET 23
23
 Mapping Process
• Draws a picture of the community
• Places resources and assets in their locations
– individual assets: skills, talent, networks,
money, etc.
– civic assets: faith associations, clubs, social
groups, etc.
– institutional assets: businesses, schools,
health services, public transportation, etc.
– environmental assets: parks, roads, farmland,
housing, clean air and water, etc.
IPDET 24
24
 Uses of Mapping
• Global Positioning System (GPS)
• Google Earth

IPDET 25
25
 Toolkit 2: Available Data
• Example sources:
– files/records
– computer data bases
– government reports
– other reports or prior evaluations
– census data
– documents (budgets, organizational charts,
policies and procedures, maps)
IPDET 26
26
 Using Agency Records
• Agencies may have already collected the
data you need
• May also have summaries and/or reports,
such as:
– internal management reports
– budget documents
– reports to the public or funding agencies
• Key issues:
– validity, reliability, accuracy

IPDET 27
27
 Advantage/Challenge:
Available Data
Advantages Often less expensive and faster
than collecting the original data
yourself
Challenges There may be coding errors or
other problems. Data may not be
exactly what is needed. You
may have difficulty getting
access. You have to verify
validity and reliability of data
IPDET 28
28
 Toolkit 3: Observation
• See what is happening
– traffic patterns
– land use patterns
– layout of city and rural areas
– quality of housing
– condition of roads
– conditions of buildings
– who goes to a health clinic

IPDET 29
29
 Ways to Observe
Unobtrusive No one knows you are observing

Participant You actually participate in the

activity
Obtrusive The people being observed know
you are there to observe them

IPDET 30
30
 Guidelines for Planning
Observations
• Develop a checklist to rate your observations
• Develop a rating scheme
• Have more than one observer, if feasible
• Train observers so they observe the same
things
• Pilot test the observation data collection
instrument
• For less formal approach, have a few key
questions in mind when you arrive
IPDET 31
31
 Advantages and
Challenges: Observation
Advantages Collects data on actual vs. self-
reported behavior or perceptions. It is
real-time vs. retrospective

Challenges Observer bias, potentially unreliable;

interpretation and coding challenges;
sampling can be a problem; can be
labor intensive; low response rates

IPDET 32
32
 Toolkit 4: Surveys
• Great for asking people about:
– perceptions, opinions, ideas
• Less accurate for measuring behavior
• Sample should be representative of the
whole
• Big problem with response rates

IPDET 33
33
 Methods for Surveys
• In-person interviews
• Mail /phone/Internet interviews or
surveys
• Self-administered questionnaires

IPDET 34
34
 Structures for Surveys
• Structured:
– Precisely worded with a range of pre-determined
responses that the respondent can select
– Everyone is asked exactly the same questions in
exactly the same way, given exactly the same
choices
• Semi-structured
– Asks the same general set of questions but may
leave many, it not all, of the answers open-ended

IPDET 35
35
 Structured vs.
Semi-structured Surveys
Structured harder to develop
easier to complete
easier to analyze
more efficient when working with large numbers
Semi- a little easier to develop
structured labor intensive to conduct
harder to analyze but provide a rich source of data
subject to bias in interpreting
burdensome for people to complete as a self-
administrated questionnaire
IPDET 36
36
 Advantages and
Challenges of Surveys
Advantages Best when you want to know what
people think, believe, or perceive,
only they can tell you that
Challenges People may not accurately recall their
behavior or may be reluctant to reveal
their behavior if it is illegal or
stigmatized. What people think they
do or say they do is not always the
same as what they actually do.
IPDET 37
37
 In-person Interviews
• Useful for in-depth understanding of
experiences, opinions, or descriptions
• Useful when other approaches do not
work
• Should be conversational
• Can be done individually or in groups
• Can be structured or semi-structured
IPDET 38
38
 Mail / Phone / Internet
Interviews and Surveys
• Literacy issues
• Consider accessibility
– reliability of postal service
– turn-around time
– do respondents have telephone access?
– do they have Internet access?

IPDET 39
39
 Self-administered
Questionnaires
• Written surveys that the respondent
completes
• Can be structured, semi-structured, or a
combination
• Should be short (no more than 20 min.)
• Usually include one or two open-ended
questions
– help make respondents more comfortable
IPDET 40
40
 Advantages of Interviews
• Can be structured, unstructured, or a
combination
• Can explore complex issues in depth
• Forgiving of mistakes: unclear questions
can be clarified during the interview and
changed for subsequent interviews
• Can provide evaluators with an intuitive
sense of the situation
IPDET 41
41
 Challenges of Interviews
• Can be expensive, labor intensive, and
time consuming
• May not be able to explore why people
have different viewpoints
• Selective hearing on the part of the
interviewer may miss information that
does not conform to pre-existing beliefs
• Cultural sensitivity: gender issues
IPDET 42
42
 Toolkit 5: Focus Groups
• Type of qualitative research where small
homogenous groups of people are
brought together to informally discuss
specific topics under the guidance of a
moderator
• Purpose: to elicit reliable data, not just
interesting information

IPDET 43
43
 Uses of Focus Groups
• Help develop a survey questionnaire
• Contextualize survey data
• Be used in tandem with surveys
• Be used as a separate data collection
tool

IPDET 44
44
 Focus Group Process
Step Process

1 Introduce the focus group meeting

2 Have the participants introduce themselves
3 Present the first question, it should be easy, an ice-breaker
4 Ask the main questions
5 Ask the last (summary) questions
6 Ask if there are other comments or questions
7 Write-up impressions, major issues and points of
discussion
IPDET 45
45
 Advantages and Challenges
of Focus Groups
Advantages Relatively quick and easy, may take less staff
time than in-depth, in-person interviews;
provides flexibility to make changes in
process and question; ability to explore
different perspectives; it can be fun
Challenges Analysis is time consuming, participants
might be different from rest of population; risk
of bias in interpreting data; risk of group being
influenced by moderator or dominant
members
IPDET 46
46
 Toolkit 6: Diaries, Journals,
Self-Reported Checklists
• Use when you want to capture
information about events in people’s
daily lives
• Participants capture experiences in real-
time not later in a questionnaire
• Used to supplement other data
collection

IPDET 47
47
 Guidelines for Diaries or
Journals
Step Process

1 Recruit people face-to-face

• encourage participation, appeal to altruism, assure
confidentiality, provide incentive
2 Provide a booklet to each participant
• cover page with clear instructions, definitions, example
• short memory-joggers, calendar
3 Consider the time-period for collecting data
• if too long, may become burdensome
• if too short may miss the behavior or event
IPDET 48
48
 Self-reported Checklists
• Cross between a questionnaire and a
diary
• The evaluator specifies a list of
behaviors or events and asks the
respondents to complete the checklist
• Done over a period of time to capture
the event or behavior
• More quantitative approach than diary
IPDET 49
49
 Advantages and Challenges
of Diaries, Journals …
Advantages Rich data can capture the details that might
be quickly forgotten over time
Good for information on how people use time
Helps in collecting sensitive information
Supplements interviews
Challenges Requires literacy
May change behavior
Data may be incomplete or inaccurate
Poor handwriting, difficult to understand
phrases IPDET 50
50
 Toolkit 7: Expert Judgment

• Interviews with experts, one-on-one or

in a panel
• Can be structured or unstructured
– everyone answers a set of specific
questions
– or free flowing (focus group model)
– or formal presentations

IPDET 51
51
 Selecting Experts
• Establish criteria for selecting experts
based on:
– recognized expert
– areas of expertise
– diverse perspectives
– diverse political views
– diverse technical expertise

IPDET 52
52
 Advantages and Challenges
of Expert Judgment
Advantages Fast, relatively inexpensive
Communication of final results can
give credibility
Challenges Weak for impact evaluation
May be based mostly on perceptions
Worth of data is only as good as the
perceived credibility of the experts

IPDET 53
53
 Toolkit 8: Delphi Technique

• Enables experts who live in different locations

to engage in a dialogue
• Experts asked specific questions
• Answers are returned to a central source for
the evaluator to summarize and feed it back
to the experts for further comments
• No one knows who said what so conflict is
avoided
• Experts can agree or argue with others’
comments
IPDET 54
54
 Advantages and Challenges
of Delphi Technique
Advantages Inexpensive
Conducive to independent thinking
Allows sharing of information
Challenges Judgments of a selected group only
Tendency to eliminate extreme positions
Time consuming and requires skill in
communication
Requires adequate time and participant
commitment
IPDET 55
55
 Toolkit 9: Citizen Report
(Score) Cards
• Used to collect citizen feedback on public
services from actual users of a service
• Can assess the performance of individual
service providers and/or compare
performance across providers
• Can generate a database of feedback on
services that can then be placed in the public
domain

IPDET 56
56
 Advantages and Challenges
of Citizen Report Cards
Advantages Mixes focus groups and
questionnaire data collection
Increases response rates
Simple communication
Challenges Local conditions must be conducive
Requires a large sample
Lack of predictability in how different
players respond
IPDET 57
57
 Developing the Survey
• Define the purpose and objectives
• Decide the important issues to be covered
• Establish the relative weight of the different
modules in the survey
• Identify important issues within sectors
• Question writers may need to learn more
about how specific programs work
• Once this background work is done, the
actual writing of the survey may begin
IPDET 58
58
 Writing Questions
• Progressively more detail is needed at
each level
– Define overarching objectives
– Balance between sectors
– Balance within sectors
– Write questions to study specific issues or
programs

IPDET 59
59
 The Right People to
Develop Questions
• Small group
– will know what subjects are of interest
(policy and analytic) to the evaluation
– will have experience using data from
similar surveys on a variety of topics
• Team must get input from policymakers
and program managers
• Use people outside to check for jargon
IPDET 60
60
 Field (Pilot) Testing
• Test with small number of subjects,
diverse areas and socioeconomic
groups
• Field test should look at three levels:
– as a whole
– each section
– individual questions
• Make revisions based on results of test
IPDET 61
61
 Sequencing Questions
• Get the respondents involved in the interview
as soon as possible
• Before asking about controversial matters,
first ask about some facts
• Intersperse fact-based questions throughout
the interview
• Ask questions about the present before
questions about the past or future
• The last questions might be to allow
respondents to provide any other information
IPDET 62
62
 General Guidelines for
Conducting Surveys (1 of 3)
• Keep it simple, clear, easy, short
• Locate other people who have done the kind
of evaluation you are interested in and locate
surveys similar to what you think you want to
do
• Make sure people know why you are asking
them to participate
• Ask questions that are easy to answer and do
not frustrate respondent’s desire to be clear in
their responses
(continued on next slide) IPDET 63
63
 General Guidelines for
Conducting Surveys (2 of 3)
• Do not ask respondents for information that
requires them to go to a file or other source. If
you must do this, you need to let them know
in advance so the material can be assembled
prior to administrating the survey.
• Respect their privacy. Treat surveys
confidentially and have procedures in place to
assure privacy. Make sure you can insure
confidentiality. Never promise anonymity
unless it can be absolutely delivered.
(continued on next slide)
IPDET 64
64
 General Guidelines for
Conducting Surveys (3 of 3)
• Respect respondents’ time and
intelligence
• Tell them how they were selected and why
their participation is important
• Do no harm: keep responses anonymous.
For example, in your report, use aggregate
responses; and assign an identification
number to the data and destroy the link to
the person’s name
IPDET 65
65
 Interviewing
• Use trained data collectors
• Use as few data collectors as possible
• Establish a protocol for data collection
• Do a “walk through” or trial run
– 3-4 face-to-face interviews
– 10 questionnaires

IPDET 66
66
 Developing an Interview
Step Procedure
1 Define the purpose of the interview. Link your purpose
to the evaluation objectives
2 Decide on open-ended or close-ended questions
3 Draft interview questions and sequence the questions
4 Prepare an Introduction and Closure for the interview
5 Prepare to record responses
6 Pre-test the instrument

IPDET 67
67
 Conducting Interviews
• Let the interviewees know:
– why they are being interviewed
– how they were selected
– how the data will be used
– whether it is confidential
– how long the interview will take
– whether you might want to talk to them
again
IPDET 68
68
 Conducting Interviews:
Set Up
• Try to pick a time a place that is quiet and
free of distractions
• Ideally, have a second person to help take
notes
• Tape recording might be a possibility; make
sure you check with interviewee and get
permission before recording
– might want to ask when setting up the appointment

IPDET 69
69
 Interviewing Skills
• Stick to your script
– if asking close-ended questions, ask
exactly the way written
– have a script for clarifications
– if asking open-ended questions, go with the
flow rather than directing it
• Be aware of cultural norms: eye contact,
direct questions, gender issues
IPDET 70
70
 More Interviewing Skills
• Balance: if you ask about what they
thing are the major supports, follow with
what you think are the major barriers
• Try to avoid asking “why” questions,
some may find this aggressive or critical
• Accept whatever they say with empathy
and without judgment

IPDET 71
71
 Conducting Interviews:
Added Touches
• Share interview questions ahead of time
– no surprises
• Offer to share a summary of what you
understand from the interview
– this might be especially useful to give the
interviewee (especially if a high ranking
official) a greater feeling of control
• Thank you note afterwards
IPDET 72
72
 Interviewing: Taking
Notes
• Take good notes without distracting from the
conversation
– maintain eye contact while writing
– write key words or phrases, not verbatim
– but — if someone is saying something you want to
capture, it is OK to ask them to repeat it or to finish
what you are writing before asking the next
question
– may want to ask “May I use your exact words?”

IPDET 73
73
 Writing-up the Interview
• Every word and idea is valuable
• Take time to write up your notes as
carefully and in-depth as possible
• It is best to do at least a brief clean-up
of notes immediately afterwards (leave
an hour between interviews)
• Write up full notes within a day of the
interview: memory decay sets in quickly
IPDET 74
74
 Writing Questionnaire
Surveys
• Must be understandable to everyone
• If possible use an existing questionnaire as a
guide
• Make the survey easy to complete
• Ask general questions first, then
demographic, then more specific, then a final
open-ended question or two
• Have draft questions reviewed by experts
• Pre-test, pre-test, pre-test
IPDET 75
75
 Questionnaires Tips and
Tricks
• Use simple, clear language, appropriate for
audience
• Ask only one question at a time
• Write your questions so that all feel their
responses are acceptable
• When possible, write questions so that
responses range from negative to positive
(continued)

IPDET 76
76
 Questionnaires Tips and
Tricks (cont.)
• Avoid “yes” or “no” responses
• Avoid absolutes at either end of the
scale (always, never), consider using
scales
• Ask questions about the current
situation
• Leave exits (no opinion)
• Avoid using double negatives
IPDET 77
77
 Planning and Logistics
• Need several weeks to plan
• Focus group sessions usually 1 to 2 hours
• Some up to 3-5 hours with planned break
(executives, in-house management, high-
level participants)
• Have clear starting and stopping times
• Consider need for food and transportation

IPDET 78
78
 Facilities and Materials
• Ideal – commercial facility designed
specifically for focus groups
• Neutral, accessible, hotel meeting room,
school or church meeting area
• Table and chairs (comfortable setting)
• Name tents
• Laptop computer for taking notes (if possible)
• Consider need for video or audio tape
recordings or have additional people take
notes
IPDET 79
79
 Number of Focus Group
Sessions
• No fixed rule
• Generally, do sessions until no new
issues emerge or when common
themes are consistent
– usually 3 to 6 group sessions

IPDET 80
80
 Ground Rules
• “What is said here, stays here”
• Everyone is encouraged to participate
but not everyone has to answer every
question
• Respect different viewpoints
• There are no right or wrong answers
• One person speaks at a time
IPDET 81
81
 Typical Questions
• What did you learn at the teachers
college that is most helpful to you in
teaching primary school?
• What are your greatest challenges in
teaching primary school students?
• What should the college teach so that
graduates are prepared to meet the
current challenges?
IPDET 82
82
 Facilitator/Moderator Role
• Be familiar with the script, rather than reading
it, so the session appears conversational
• Make sure everyone is heard
• Manage time, closing off discussion, moving
to next topic
• Set ground rules
• Say as little as possible
• Keep personal views outside the room
• Use active listening
• Probe for elaboration (Tell me more)
IPDET 83
83
 Write-up
• Write-up your impressions immediately after
each focus group
– major issues, major points of discussion
– anything unusual that happened
• Compare with partner
• If recorded, make arrangements to have it
transcribed, or to carefully watch/listen to it
later
• Identify themes and emerging issues

IPDET 84
84
 Data Collection Summary
• You can choose more than one data
collection technique, multiple tools often
help you meet the evaluation needs
• Do not let the tool drive your work
• Choose the correct tool to meet the
needs of the evaluation

IPDET 85
85
To continue on to the
Next Module click here
To return to the
Table of Contents click here