Northwest Center for Public Health Practice

Data Collection for Program Evaluation

Introduction
This toolkit offers some additional information, templates, and resources to assist
you in planning your own data collection for program evaluation.

General Evaluation

Program evaluation is a systematic way to collect information about the

characteristics, activities, and results of a program in order to make decisions about the program. Evaluating a program helps you determine
whether it is functioning as intended, or meeting its goals and objectives, and may help you identify areas for improvement. At NWCPHP
we use the CDCs Evaluation Framework to guide our practice. The
CDC Evaluation Framework is described in our online module Program
Evaluation in Public Health. Information about the framework and
related resources are at www.cdc.gov/eval/framework.htm#summary.

Steps
Engage
stakeholders
Ensure use
and share
lessons learned

Justify
conclusions

Resources
Introduction to Program Evaluation for Public Health Programs The CDC
also offers a 92-page self study guide, which includes worksheets and checklists
for implementing the steps in the framework.
www.cdc.gov/eval/evalguide.pdf
Public Health Agency of Canadas Evaluation Toolkit includes specific information about how to plan for, design, conduct, and use the results of program
evaluations. The toolkit is divided by topic area, and includes many worksheets
and tools that you can print out for use in your organization.
www.phac-aspc.gc.ca/php-psp/toolkit-eng.php
The site also has blank worksheets to help you establish your evaluation questions and your data collection plan, identify stakeholders, and interpret your
findings.
www.phac-aspc.gc.ca/php-psp/pdf/toolkit/Appendix%20B.pdf
The Practice of Health Program Evaluation (2001) For more advanced evaluators, or for people interested in further study, Dr. David Grembowskis book
provides a thorough and academic discussion of program evaluation. The chapters most closely related to this course are: Chapter 3Developing Evaluation
Questions, Chapter 8Measurement and Data Collection, Chapter 9Data
Analysis, Chapter 10Disseminating the Answers.

Standards
Utility
Feasibility
Propriety
Accuracy

Gather credible
evidence

Describe
the program

Focus the
evaluation
design

Data Collection for Program Evaluation

Example
Evaluation Plan
Program:
Date:
Evaluation
Question

Indicators

Data Source/
Method

Person Responsible

Word version of sample template

Overview of Data Collection

This course focuses on step 4 of the CDC Framework: Gather Credible Evidence.
There are many different methods for gathering data. You should select the
method that best suits your needs.

Resources
Kellogg Foundations Evaluation Handbook describes data collection methods
in more detail than this course was able to cover.
www.wkkf.org/pubs/tools/evaluation/pub770.pdf (see pages 6996)
The Power of Proof: An Evaluation Primer provides information about preparing to collect data, the different methods for collecting data, as well as tips for
best practice. While this resource is designed for evaluating tobacco prevention
and cessation programs, it is applicable to other areas of public health practice.
www.ttac.org/power-of-proof
www.ttac.org/power-of-proof/data_coll

Institutional Review Boards

When you design your program evaluation, it is important to consider whether

you need to contact an Institutional Review Board (IRB). IRBs are found at most
universities and other large organizations that receive federal funding (such
as hospitals, research institutes, and public health departments). An IRB is a
committee of people who review proposed projects to ensure that the principles
of autonomy, beneficence, and justice are honored.

Timeline

Data Collection for Program Evaluation

It is a fine line between evaluation and research, so it is important that you

consider human subject protections every time your evaluation involves observations of people, interviews, surveys, or the collection of peoples personal
health information. The Washington State Department of Health developed
the decision tree below to illustrate the difference between research and nonresearch. In general consult an IRB if
Your evaluation involves getting information from or about people
Your institution or any of your collaborators receive federal funds
You hope that the findings of your evaluation can inform other programs

Is It Research or Public Health Practice?

What is the primary intent?

If the project involves human

subjects, is intended to generate
or contribute to generalizable
knowledge to improve public health
practice, and the benefits extend to
society, then the project is research.

Contact your agencys designated

IRB and describe the proposed
project in detail.

If the project is intended to prevent a disease or injury and

improve health (or to improve a current, on-going public health
program or service) and it benefits primarily the participants,
it may be non-research.

The project is an
evaluation of a new, modified,
or previously untested intervention, service, or program.

The project involves data

collection/use for the purpose of
assessing community health status
or evaluating an established programs
success in achieving its objectives in
a specific population.

Does the project involve vulnerable populations or collect sensitive/

personal information?

Unless IRB staff determine the

project meets criteria for
exemption, IRB review
will be required.

If yes, informal
consultation
with WSIRB is
recommended.
360.902.8075.

If no, complete
the Questions to
Consider
When Using
Human
Participants in
Public Health
Assessment &
Evaluation and
follow your
departments
protocols.

Data Collection for Program Evaluation

They will help you determine whether you are doing research, whether your
research actually involves human subjects, and whether your research may be
exempt from human subjects regulations due to lack of risk to participants.

Resources
Washington State Department of Health Human Subjects Guide
www.doh.wa.gov/Data/Guidelines/HumanSubjectsguide.htm
University of Washington Human Subjects Division
www.washington.edu/research/hsd/index.php
UW Human Subjects Division FAQ
www.washington.edu/research/hsd/faq.php
The Belmont Report discusses U.S. law related to ethical treatment of human
subjects.
ohsr.od.nih.gov/guidelines/belmont.html

Data Collection for Program Evaluation

Data Collection Methods

Method

Use when

Advantages

Disadvantages

Document Review

Program documents or
literature are available and
can provide insight into the
program or the evaluation

Data already exist

Does not interrupt the
program
Little or no burden on
others
Can provide historical or
comparison data
Introduces little bias

Time consuming
Data limited to what exists
and is available
Data may be incomplete
Requires clearly defining
the data youre seeking

Observation

You want to learn how the

program actually operatesits
processes and activities

Allows you to learn

about the program as it is
occurring

Time consuming
Having an observer can
alter events

Can reveal unanticipated

information of value
Flexible in the course of
collecting data

Difficult to observe multiple

processes simultaneously
Can be difficult to interpret
observed behaviors

Survey

You want information directly

from a defined group of
people to get a general idea
of a situation, to generalize
about a population, or to get
a total count of a particular
characteristic

Many standardized instruments available

Can be anonymous
Allows a large sample
Standardized responses
easy to analyze
Able to obtain a large
amount of data quickly
Relatively low cost
Convenient for respondents

Sample may not be

representative
May have low return rate
Wording can bias responses
Closed-ended or brief
responses may not provide
the whole story
Not suited for all people
e.g., those with low reading
level

Interview

You want to understand

impressions and experiences
in more detail and be able to
expand or clarify responses

Often better response rate

than surveys
Allows flexibility in
questions/probes
Allows more in-depth information to be gathered

Time consuming
Requires skilled interviewer
Less anonymity for
respondent
Qualitative data more difficult to analyze

Focus Group

You want to collect in-depth

information from a group of
people about their experiences and perceptions related
to a specific issue.

Collect multiple peoples

input in one session
Allows in-depth discussion
Group interaction can
produce greater insight
Can be conducted in short
time frame
Can be relatively inexpensive compared to interviews

Requires skilled facilitator

Limited number of questions can be asked
Group setting may inhibit
or influence opinions
Data can be difficult to
analyze
Not appropriate for all
topics or populations

Data Collection for Program Evaluation

Document Review

When evaluating a program it is helpful to review existing documents to gather

information. You can review meeting minutes, sign in sheets, quarterly or annual
reports, or surveillance data to learn more about the activities of the program
and its reach. You can also review related scientific literature or Web sites to
learn how other similar programs work or what they accomplished. This can
help inform your evaluation design.

Resources
CDC Evaluation Brief has more information about using existing documents to
collect data for program evaluation.
www.cdc.gov/healthyyouth/evaluation/pdf/brief18.pdf

Example
Practical Evaluation: Beginning with the End
Session III: Giving Data Meaning
Type and source of data for possible use in practical evaluation or community
health assessment
Birth certificates
Death records
Disease incidence and prevalence
rates
Divorce rates
Education levels of population
(number and percentage of adults
completing highschool, dropout
rates)
Environmental health issues (water
and air quality)
English as a Second Language
Health services available in
community, proximity to hospitals,
social services, etc.
Barriers to accessing services (transportation, hours, language)
Hospital discharge data
Past decision-making in the
community

Indicators of cultural communities (community centers, theaters,

museums, festivals, dance, ethnic
heritage centers or celebrations,
cultural organizations, houses of
worship)
Indicators of social health (e.g.,
substance abuse, crime, child
abuse cases)
Leading causes of death with age,
race, and gender specific data
where possible
Leading causes of morbidity with
age, race, and gender specific data
where possible
License data
Literacy rates
Locally owned businesses
Mean family size
Median family income by race

Data Collection for Program Evaluation

Percent of community who are

Medicaid eligible
Percentage of households who rent
or own their homes
Persons below the poverty level
(number and percentage) by race
Poison control center data
Population distributed by age, race,
and gender

Registry data
Single heads of household (number
and percentage)
Surveillance data
Transportation issues related to
healthcare (indicator = miles of
public transit per capita)
Un/employment rates by race
Vacant land

Property assessments
Sources
Minkler, M. (Ed.), (1997). Community Organizing & Community Building for Health. New
Brunswick, NJ: Rutgers University Press.
Quinn, S., (1997). Unpublished syllabus. Chapel Hill, NC: University of North Carolina.

Observation

Observation gathers information about a program as the programs activities

occur. Examples could be observing services being provided, training sessions,
meetings, or special events. Observation done in an unobtrusive manner can
provide important information about what really takes place.

Resources
Collecting Evaluation Data: Direct Observation The University of Wisconsin
Extension published a number of brief summaries about program evaluation
and methods for evaluation. This segment has sample observation checklist
templates, a list of aspects of programs that can be systematically observed, and
sample field notes.
learningstore.uwex.edu/pdf/G3658-5.pdf
The Power of Proof also offers a relatively brief overview of how, when, and
why you might use observation for evaluation.
www.ttac.org/power-of-proof/data_coll/observation

Example
As we discussed in the course, its important to decide what you need to
observe before you collect data by observation. It is helpful to make a checklist
of things you need to look for during the observation period. This is the observation checklist that Anita developed to assess the Brief Preparedness Assessment
and Intervention.

Data Collection for Program Evaluation

BPAI Observation
Checklist

Patient visit 1
( done)

Patient visit 2
( done)

Patient visit 3
( done)

Patient visit 4
( done)

Patient visit 5
( done)

Awareness

Planning

Actions taken

Emergency/disasters

Local/county
response

Personal planning

Supplies/equipment

Preparedness Plan
booklet

What to Do booklet

High

Medium

Low

Assessment

Intervention (key points)

Written materials provided

Public Info Program

card
Emergency phone
number card
Patient interest level

Total time
Notes

Word version of checklist

Data Collection for Program Evaluation

Surveys

Surveys allow data to be collected from a large group of people and, depending
upon how the survey is administered, can allow people to remain anonymous.
Survey instruments or questionnaires ask questions in a standardized format that
allows consistency and the ability to aggregate responses. Potential questions can
focus on the collection of qualitative or quantitative data.

Resources
Social Research Methods is a useful site for learning about surveys, interviews,
and focus groups.
www.socialresearchmethods.net
The Survey Research section is most relevant to surveys, interviews, and focus
groups.
www.socialresearchmethods.net/kb/survey.php
Collecting Evaluation Data: Surveys explores reason to use surveys, alternative to surveys, survey methods (e.g., self-administered, mail, telephone,
Web-based), advantages and disadvantages of each, and survey planning and
implementation. It also includes a sample cover letter, follow-up post card, and
press release.
learningstore.uwex.edu/pdf/G3658-10.pdf
Questionnaire DesignAsking Questions with a Purpose covers the pros
and cons of questionnaire use, example questions, a comprehensive formatting
guide, and a reference list.
learningstore.uwex.edu/pdf/G3658-2.pdf

Write more effective survey questions

Pamela Narins, Manager, Market Research
Ryerson University, Toronto

Tips and rules to help you improve your survey question writing
Naturally, no question is good in all situations, but there are some general
rules to follow. Using these rules and examples will help you write useful
questions.
1. Remember your surveys purpose
All other rules and guidelines are based on this one. There was a reason you
decided to spend your time and money to do your survey, and you should
ensure that every question you ask supports that reason. If you start to get lost
while writing your questions, refer back to this rule.
2. If in doubt, throw it out
This is another way of stating the first rule, but it is important enough to
repeat. A question should never be included in a survey because you cant
think of a good reason to discard it. If you cannot come up with a concrete
research benefit that will result from the question, dont use it.

Data Collection for Program Evaluation

3. Keep your questions simple

Compound sentences force respondents to keep a lot of information in their
heads, and are likely to produce unpredictable results. Example: Imagine
a situation where the production supervisor is away from the line, a series
of defective parts is being manufactured, and you just heard that a new
client requires ten thousand of these parts in order to make their production
schedule. How empowered do you feel by your organization to stop the line
and make the repairs to the manufacturing equipment? This question is too
complex for a clear, usable answer. Try breaking it down into component
parts.
4. Stay focusedavoid vague issues
If you ask When did you last see a movie? you might get answers that refer
to the last time your respondent rented a video, when you are really interested in the last time the respondent went out to a movie theater. Consider
too, Please rate your satisfaction with the service you have received from
this company. This is a fine general question, but will not likely lead to
any specific action steps. Particular elements of service must be probed if
responses are to result in specific recommendations.
5. If a question can be misinterpreted, it will be
What time do you normally eat dinner? will be answered differently by
people living in different regions; dinner can refer to either the midday or
the evening meal. Be clear, concise, always beware of imprecise language
and avoid double negatives.
6. Include only one topic per question (avoid double-barreled questions)
How would you interpret the responses to Please rate your satisfaction with
the amount and kind of care you received while in the hospital. or, a question asking about speed and accuracy? If you want to be able to come up
with specific recommended actions, you need specific questions.
7. Avoid leading questions
It is easy, and incorrect, to write a question that the respondent believes has
a right answer. Most doctors believe that exercise is good for you. Do you
agree? is an example of a leading question. Even the most well-meaning
researcher can slant results by including extraneous information in a question.
Leading questions can be used to prejudice results.
8. Consider alternate ways to ask sensitive questions
Some questions are obviously sensitive. Income, drug or alcohol consumption and sexual habits are clear examples of topics that must be asked about
carefully. The question: Did you vote in the last election? has an element
of sensitivity in it as well. Respondents might be unwilling to admit that they
did not vote, because of civic pride or embarrassment. To avoid respondent
alienation, it can be useful to mitigate the cost of answering No by including a way out. For example: There are many reasons why people dont get a
chance to vote. Sometimes they have an emergency, or are ill, or simply cant
get to the polls. Thinking about the last election, do you happen to remember

10

Data Collection for Program Evaluation

if you voted? Also, people are less likely to lie about their age in face-to-face
interviews if they are asked what year they were born, rather than how old
they are.
9. Make sure the respondent has enough information
Asking respondents How effective has this companys new distribution
program been? may not be as effective as Recently, we implemented a new,
centralized distribution system. Did you know this? Followed by Have you
seen any positive benefits resulting from this change? It can be beneficial to
break down questions that require background information into two parts: a
screening item describing the situation which asks if the respondent knows
about it, and a follow-up question addressing attitudes the respondent has
about the topic.
Five rules for obtaining usable answers
Useful answers are just as important as good questions. Here are some rules:
1. Response options need to be mutually exclusive and exhaustive
This is the most important rule to follow when providing response options. If
response options are not mutually exclusive, the respondent will have more
than one legitimate place for their answer. The response choices, 1 to 2, 2
to 3 and More than 3 pose a problem for someone whose answer is 2.
You must also ensure that the response options you provide cover every
possibility. Asking Which of the following beverages did you drink at least
once during the past seven days? and providing a list of coffee, soda and tea
might be sufficient if you were doing a study on the consumption of caffeinated drinks. But, they would not work if you wanted to know about broader
consumption habits. If you are unable to provide a complete list of options,
at least provide an Other choice. If the list of choices is too long, an open
ended-question might be a better option.
2. Keep open-ended questions to a minimum
While open-ended (or verbatim) questions are a valuable tool, they should
not be over-used. Not only can they result in respondent fatigue, but they
pose problems in terms of coding and analysis.
3. People interpret things differently, particularly when it comes to time
Trouble-spots include responses such as Always, Sometimes and Never.
You must build in a temporal frame of reference to ensure that all respondents are answering in the same way. As in this example from an intervieweradministered questionnaire, I am going to read a list of publications. For
each one, please tell me whether you read it regularly. By regularly I mean, at
least three out of every four issues.
4. Consider a Dont Know response
It is useful to allow people to say they simply do not have an opinion about a
topic. However, some investigators worry that people will opt for that choice,
reducing the ability to analyze responses. Evidence shows that this fear is

11

Data Collection for Program Evaluation

largely unfounded. The goal of your research should help you decide if a
Dont Know option would be wise. For example, if you only want information from those with an informed opinion or higher interest, offer a Dont
Know choice.
5. Provide a meaningful scale
The end points of response scales must be anchored with meaningful labels.
For example, Please rate your satisfaction with customer service. Lets use
a scale where 1 means Very Satisfied and 5 means Very Dissatisfied. You
could also give each point on the scale a label. The number of scale points
(3, 5 or 7) can have little effect on the conclusions you draw later. Choosing
how many points, then, is often a matter of taste. There are three things to
remember when constructing a response scale. First, an odd number of points
provides a middle alternative. This is a good way to provide respondents with
moderate opinions a way out (similar to the Dont Know, choice above).
Secondly, if measuring extreme opinions is critical, use a scale with a greater
number of points. Finally, you generally gain nothing by having a scale with
more than 7 points and will probably find that you will collapse larger scales
when it comes time to analyze the data.
The price of poorly written questions
Well-written questions are critical. Participants must stay interested. If your
respondents start to feel alienated by threatening, emotional or difficult questions, response rates are likely to go down and response bias will probably go up.
Also, respondents can get frustrated if your questions do not provide answer
choices that match their opinions or experiences. The quality of your collected
data will suffer; your analyses will be less meaningful; and the whole research
process may prove useless or harmful. So think carefully about the questions
you write, look at reputable examples of questions, and refer to the rules above.
If you follow these guidelines, youll do fine.

Examples
Questionnaire template in Word

12

Data Collection for Program Evaluation

Interviews

Conducting interviews is a method that, like open-ended questions in a questionnaire, allows you to obtain an individuals response in their own words.
Interviews differ from questionnaires in that they elicit more detailed qualitative data and allow you to interact with the person to better understand
their response. Interviews may be conducted in-person or over the phone.
Interviewing is useful when you want more in-depth information about a
persons attributes, knowledge, attitudes/beliefs, or behaviors.

Resources
Key Informant Interviews by the University of Illinois Extension provides an
excellent resource for learning more about key informant interviews.
ppa.aces.uiuc.edu/KeyInform.htm
Research Methods Knowledge Base provides a great introduction to
interviewing.
www.socialresearchmethods.net/kb/intrview.php

Examples
Interview Documentation Word template

Focus Groups

Like an interview, a focus group allows you to collect qualitative data. However,
unlike interviews, in which data are collected by one-on-one interactions, focus
groups provide data about a particular topic through small group discussions.
Focus groups are an excellent method for obtaining opinions about programs
and services. They produce information from many people in a short period of
time, so can be an effective method when information is needed quickly.

Resources
Performance Monitoring and Evaluation Tips is a very simple four-page guide
to conducting focus groups.
www.usaid.gov/pubs/usaid_eval/pdf_docs/pnaby233.pdf
Using Focus Groups by the University of Toronto Health Communication Unit
is a more thorough review of focus group design and use.
www.thcu.ca/resource_db/pubs/982989842.pdf

13

Data Collection for Program Evaluation

Sampling

When you collect data, you should think about your sample. Who will you
recruit to complete your questionnaire or participate in a focus group? How will
you recruit participants? How many should you recruit? As we discussed in the
course, some of the answers to these questions depend on the sort of information you need.

Resources
Sampling and Sample Size Guide / Logistics Guides by the Public Health
Agency of Canada offers an excellent and brief description of sampling
techniques.
www.phac-aspc.gc.ca/php-psp/pdf/toolkit/Appendix%20D%201-3.pdf
Sampling, a 12-page guide by University of Wisconsin Extension, has a table of
random numbers and suggested sample sizes needed to detect change.
learningstore.uwex.edu/pdf/G3658-03.pdf

Data Analysis
Resources
Analyzing Qualitative Data
learningstore.uwex.edu/pdf/G3658-12.pdf
Analyzing Quantitative Data
learningstore.uwex.edu/pdf/G3658-6.pdf
Problems With Using Microsoft Excel for Statistics explains why Microsoft
Excel should not be used for more complex statistical analysis.
www.stat.uiowa.edu/~jcryer/JSMTalk2001.pdf

Other Helpful Resources

SMART objectives
www.marchofdimes.com/files/HI_SMART_objectives.pdf
www.cdc.gov/healthyyouth/evaluation/pdf/brief3b.pdf
Reading levels
school.discoveryeducation.com/schrockguide/fry/fry.html

14