IPDET

Module 8:
Selecting and Constructing
Data Collection
Instruments
 Introduction

• Data Collection Strategies

• Characteristics of Good Measures
• Quantitative and Qualitative Data
• Tools for Collecting Data

22
 Data Collection
Strategies
• No one best way: decision depends on:
– What you need to know: numbers or stories
– Where the data reside: environment, files,
people
– Resources and time available
– Complexity of the data to be collected
– Frequency of data collection
– Intended forms of data analysis

33
 Rules for Collecting
Data
• Use multiple data collection methods
• Use available data, but need to know
– how the measures were defined
– how the data were collected and cleaned
– the extent of missing data
– how accuracy of the data was ensured

44
 Rules for Collecting
Data
• If must collect original data:
– be sensitive to burden on others
– pre-test, pre-test, pre-test
– establish procedures and follow them
(protocol)
– maintain accurate records of
definitions and coding
– verify accuracy of coding, data input
55
 Structured Approach
• All data collected in the same way
• Especially important for multi-site
and cluster evaluations so you can
compare
• Important when you need to make
comparisons with alternate
interventions

66
 Use Structured Approach
When:

• need to address extent questions

• have a large sample or population
• know what needs to be measured
• need to show results numerically
• need to make comparisons across
different sites or interventions

77
 Semi-structured
Approach
• Systematic and follow general
procedures but data are not
collected in exactly the same way
every time
• More open and fluid
• Does not follow a rigid script
– may ask for more detail
– people can tell what they want in
their own way
88
 Use Semi-structured
Approach when:
• conducting exploratory work
• seeking understanding, themes,
and/or issues
• need narratives or stories
• want in-depth, rich, “backstage”
information
• seek to understand results of data
that are unexpected
99
 Characteristics of
Good Measures
• Is the measure relevant?
• Is the measure credible?
• Is the measure valid?
• Is the measure reliable?

11
00
 Relevance
Does the Do not measure
measure what is easy
capture what instead of what
matters? is needed

11
11
 Credibility

Is the measure believable? Will it

be viewed as a reasonable and
appropriate way to capture the
information sought?

11
22
 Internal Validity
How well does Are waiting
the measure lists a valid
capture what it measure of
is supposed to? demand?

11
33
 Reliability
A measure’s How reliable are:
precision and – birth weights of
stability- extent newborn
to which the infants?
same result – speeds
would be measured by a
obtained with stopwatch?
repeated trials

11
IPDET © 2009 44
 Quantitative Approach
• Data in numerical form
• Data that can be precisely
measured
– age, cost, length, height, area,
volume, weight, speed, time, and
temperature
• Harder to develop
• Easier to analyze
11
IPDET © 2009 55
 Qualitative Approach
• Data that deal with description
• Data that can be observed or self-reported,
but not always precisely measured
• Less structured, easier to develop
• Can provide “rich data” — detailed and
widely applicable
• Is challenging to analyze
• Is labor intensive to collect
• Usually generates longer reports

11
IPDET © 2009 66
 Which Data?
If you: Then Use:
- want to conduct statistical analysis
- want to be precise Quantitati
- know what you want to measure
ve
- want to cover a large group
- want narrative or in-depth information
- are not sure what you are able to measure Qualitati
- do not need to quantify the results ve
11
IPDET © 2009 77
 Obtrusive vs.
Unobtrusive Methods
Obtrusive
data collection Unobtrusive
methods that data collection
directly obtain methods that do not
information from collect information
directly from
those being
evaluees
evaluated
e.g., document
e.g. interviews, analysis, GoogleEarth,
surveys, focus groups observation at a
distance, trash of the
stars
11
IPDET © 2009 88
 How to Decide on Data
Collection Approach
• Choice depends on the situation
• Each technique is more
appropriate in some situations
than others
• Caution: All techniques are subject
to bias

11
IPDET © 2009 99
 Triangulation to
Increase Accuracy of
Data
• Triangulation of methods
– collection of same information using
different methods
• Triangulation of sources
– collection of same information from a
variety of sources
• Triangulation of evaluators
– collection of same information from more
than one evaluator
22
IPDET © 2009 00
IPDET © 2009 22
11
 Data Collection Tools
• Participatory Methods
• Records and Secondary Data
• Observation
• Surveys and Interviews
• Focus Groups
• Diaries, Journals, Self-reported Checklists
• Expert Judgment
• Delphi Technique
• Other Tools
22
IPDET © 2009 22
 Tool 1: Participatory
Methods
• Involve groups or communities
heavily in data collection
• Examples:
– community meetings
– mapping
– transect walks

22
IPDET © 2009 33
 Community Meetings
• One of the most common
participatory methods
• Must be well organized
– agree on purpose
– establish ground rules
• who will speak
• time allotted for speakers
• format for questions and answers

22
IPDET © 2009 44
 Mapping
• Drawing or using existing maps
• Useful tool to involve stakeholders
– increases understanding of the
community
– generates discussions, verifies secondary
sources of information, perceived changes
• Types of mapping:
– natural resources, social, health,
individual or civic assets, wealth, land
use, demographics
22
IPDET © 2009 55
 Transect Walks
• Evaluator walks around community
observing people, surroundings,
and resources
• Need good observation skills
• Walk a transect line through a map
of a community — line should go
through all zones of the
community
22
IPDET © 2009 66
 Tool 2: Records and
Secondary Data
• Examples of sources:
– files/records
– computer data bases
– industry or government reports
– other reports or prior evaluations
– census data and household survey data
– electronic mailing lists and discussion groups
– documents (budgets, organizational charts,
policies and procedures, maps, monitoring
reports)
– newspapers and television reports
22
IPDET © 2009 77
 Using Existing Data
Sets
Key issues: validity, reliability,
accuracy, response rates, data
dictionaries, and missing data
rates

22
IPDET © 2009 88
 Advantage/Challenge:
Available Data
Advantages Often less expensive and
faster than collecting the
original data again
Challenges There may be coding errors
or other problems. Data may
not be exactly what is
needed. You may have
difficulty getting access. You
have to verify validity and
reliability of data
22
IPDET © 2009 99
 Tool 3: Observation
• See what is happening
– traffic patterns
– land use patterns
– layout of city and rural areas
– quality of housing
– condition of roads
– conditions of buildings
– who goes to a health clinic
33
IPDET © 2009 00
 Observation is Helpful
when:
• need direct information
• trying to understand ongoing
behavior
• there is physical evidence, products,
or outputs than can be observed
• need to provide alternative when
other data collection is infeasible or
inappropriate
33
IPDET © 2009 11
 Degree of Structure of
Observations
• Structured: determine, before the
observation, precisely what will be
observed before the observation
• Unstructured: select the method
depending upon the situation with no
pre-conceived ideas or a plan on what
to observe
• Semi-structured: a general idea of what
to observe but no specific plan
33
IPDET © 2009 22
 Ways to Record
Information from
Observations
• Observation guide
– printed form with space to record
• Recording sheet or checklist
– Yes/no options; tallies, rating scales
• Field notes
– least structured, recorded in
narrative, descriptive style

33
IPDET © 2009 33
 Guidelines for Planning
Observations
• Have more than one observer, if
feasible
• Train observers so they observe the
same things
• Pilot test the observation data
collection instrument
• For less structured approach, have a
few key questions in mind

33
IPDET © 2009 44
 Advantages and
Challenges:
Observation
Advantage Collects data on actual vs. self-
s reported behavior or
perceptions. It is real-time vs.
retrospective
Challenges Observer bias, potentially
unreliable; interpretation and
coding challenges; sampling can
be a problem; can be labor
intensive; low response rates
33
IPDET © 2009 55
 Tool 4: Surveys and
Interviews
• Excellent for asking people about:
– perceptions, opinions, ideas
• Less accurate for measuring
behavior
• Sample should be representative
of the whole
• Big problem with response rates
33
IPDET © 2009 66
 Structures for Surveys
• Structured:
– Precisely worded with a range of pre-
determined responses that the respondent
can select
– Everyone asked exactly the same questions
in exactly the same way, given exactly the
same choices
• Semi-structured
– Asks same general set of questions but
answers to the questions are predominantly
open-ended
33
IPDET © 2009 77
 Structured vs.
Semi-structured
Surveys
Structure harder to develop
d easier to complete
easier to analyze
more efficient when working with large
numbers
Semi- easier to develop: open ended questions
structure more difficult to complete: burdensome for
d people to complete as a self-administrated
questionnaire
harder to analyze but provide a richer
source of data, interpretation of open- 33
IPDET © 2009ended responses subject to bias 88
 Modes of Survey
Administration
• Telephone surveys
• Self-administered questionnaires
distributed by mail, e-mail, or
websites
• Administered questionnaires,
common in the development context
• In development context, often issues
of language and translation
33
IPDET © 2009 99
 Mail / Phone / Internet
Surveys
• Literacy issues
• Consider accessibility
– reliability of postal service
– turn-around time
• Consider bias
– What population segment has
telephone access? Internet access?

44
IPDET © 2009 00
 Advantages and
Challenges of Surveys
Advantage Best when you want to know
s what people think, believe, or
perceive, only they can tell you
that
Challenges People may not accurately recall
their behavior or may be
reluctant to reveal their behavior
if it is illegal or stigmatized. What
people think they do or say they
do is not always the same as 44
IPDET © 2009
what they actually do. 11
 Interviews

• Often semi-structured
• Used to explore complex issues in
depth
• Forgiving of mistakes: unclear
questions can be clarified during the
interview and changed for
subsequent interviews
• Can provide evaluators with an
intuitive sense of the situation
44
IPDET © 2009 22
 Challenges of
Interviews
• Can be expensive, labor intensive,
and time consuming
• Selective hearing on the part of
the interviewer may miss
information that does not conform
to pre-existing beliefs
• Cultural sensitivity: e.g., gender
issues
44
IPDET © 2009 33
 Tool 5: Focus Groups
• Type of qualitative research where
small homogenous groups of
people are brought together to
informally discuss specific topics
under the guidance of a moderator
• Purpose: to identify issues and
themes, not just interesting
information, and not “counts”
44
IPDET © 2009 44
 Focus Groups Are
Inappropriate when:
• language barriers are
insurmountable
• evaluator has little control over the
situation
• trust cannot be established
• free expression cannot be ensured
• confidentiality cannot be assured
44
IPDET © 2009 55
 Focus Group Process
Phase Action
1 Openin Ice-breaker; explain purpose; ground rules;
g
introductions
2 Warm- Relate experience; stimulate group
up interaction; start with least threatening and
simplest questions
3 Main Move to more threatening or sensitive and
body complex questions; elicit deep responses;
connect emergent data to complex, broad
participation
4 Closur End with closure-type questions; summarize
e and refine; present theories, etc; invite final
44
IPDET © 2009
comments or insights; thank participants66
 Advantages and
Challenges of Focus
Groups
Advantage Can be conducted relatively quickly and
s easily; may take less staff time than in-
depth, in-person interviews; allow
flexibility to make changes in process
and questions; can explore different
perspectives; can be fun
Challenge Analysis is time consuming; participants
s not be representative of population,
possibly biasing the data; group may be
influenced by moderator or dominant
group members
44
IPDET © 2009 77
 Tool 6: Diaries and Self-
Reported Checklists
• Use when you want to capture
information about events in
people’s daily lives
• Participants capture experiences in
real-time not later in a
questionnaire
• Used to supplement other data
collection
44
IPDET © 2009 88
 Guidelines for Diaries
Step
or
Process
Journals
1 Recruit people face-to-face
• encourage participation, appeal to altruism,
assure
confidentiality, provide incentive
2 Provide a booklet to each participant
• cover page with clear instructions, definitions,
example
• short memory-joggers, explain terms, comments
on last
page , calendar
3 Consider the time-period for collecting data 44
IPDET © 2009 99
 Self-reported
Checklists
• Cross between a questionnaire and a
diary
• The evaluator specifies a list of
behaviors or events and asks the
respondents to complete the checklist
• Done over a period of time to capture
the event or behavior
• More quantitative approach than diary

55
IPDET © 2009 00
 Advantages and Challenges of
Diaries and Self-reported
Checklists
Advantage Can capture in-depth, detailed data that might be
s otherwise forgotten
Can collect data on how people use their time
Can collect sensitive information
Supplements interviews provide richer data

Challenge Requires some literacy

s May change behavior
Require commitment and self-discipline
Data may be incomplete or inaccurate
Poor handwriting, difficult to understand phrases

55
IPDET © 2009 11
 Tool 7: Expert Judgment

Use of experts, Can be structured

one-on-one or as a or unstructured
panel Issues in
E.g., Government selecting experts
task forces,
Advisory Groups

55
IPDET © 2009 22
 Selecting Experts
• Establish criteria for selecting
experts not only on recognition as
expert but also based on:
– areas of expertise
– diverse perspectives
– diverse political views
– diverse technical expertise

55
IPDET © 2009 33
 Advantages and
Challenges of Expert
Judgment
Advantage Fast, relatively inexpensive
s
Challenges Weak for impact evaluation
May be based mostly on
perceptions
Value of data depends on how
credible the experts are
perceived to be
55
IPDET © 2009 44
 Tool 8: Delphi Technique
• Enables experts to engage remotely in a
dialogue and reach consensus, often
about priorities
• Experts asked specific questions; often
rank choices
• Responses go to a central source, are
summarized and fed back to the experts
without attribution
• Experts can agree or argue with others’
comments
• Process may be iterative
55
IPDET © 2009 55
 Advantages and
Challenges of Delphi
Technique
Advantage Allows participants to remain anonymous
s Is inexpensive
Is free of social pressure, personality
influence, and individual dominance
Is conducive to independent thinking
Allows sharing of information
Challenges May not be representative
Has tendency to eliminate extreme
positions
Requires skill in written communication
Requires time and participant
55
commitment
IPDET © 2009 66
 Other Measurement
Tools
- scales (weight) - health testing
- tape measure tools:
- stop watches i.e. blood pressure

- chemical tests : - aptitude and

achievement
i.e. quality of
water tests
-citizen report
cards
55
IPDET © 2009 77
 Data Collection
Summary
Choose more than one data
collection technique
No “best” tool
Do not let the tool drive your work
but rather choose the right tool to
address the evaluation question

55
IPDET © 2009 88
 A Final Note….
“I never guess. It is a capital mistake
to theorize before one has data.
Insensibly one begins to twist facts and
theories,
instead of theories to suit facts.”
--Sir Arthur Conan Doyle

Question
s?
55
IPDET © 2009 99