Public Health Action & Evaluation

Evaluation in Public Health

Dr. Tafadzwa Nyanhanda

Week 9

1
 Overview
➢Role of evaluation in public health
➢Types of evaluation (formative, process, outcomes)
➢Insiders vs outsiders in evaluation
➢Importance of stakeholders
➢Developing aims and objectives

2
 What is evaluation?
“The systematic collection of information about the
activities, characteristics, and outcomes of programs
to make judgments about the program, improve
program effectiveness, and/or inform decisions about
future program development.”
CDCs Introduction to Program Evaluation for Public Health Programs

Done for and with specific intended primary users for

specific, intended uses.

Public Health initiatives or interventions aim to create social and physical environments
that promote good health.
Public Health initiatives or interventions should be assessed to determine if it is making
an impact in improving the quality of life of people and communities through the
elimination or the reduction in the incidence, prevalence, and rates of disease, and
disability.

Done for and with specific intended primary users for specific, intended uses e.g. for
accountability – to clients, community, funders, social justice – to ensure vulnerable
populations receive appropriate and effective services

3
 Evaluation vs Research
Evaluation Research
• Results not generalizable • Results generalizable
• Designed to improve • Prove cause-and-effect
initiative – decision making relationship (controlled
• Process (why, how well an situation)
intervention worked, did the • Endpoint
intervention work?)
• Researcher’s interest/agenda
• Evaluation questions
determined by stakeholders

Before we talk about what evaluation is, let’s review what evaluation isn’t.
Although there is some overlap between research and evaluation, there is need to point
out the some differences between evaluation and research which are outlined in the
slide.

4
 The purposes of evaluation
➢ To assess the extent to which a program is being
implemented
➢ To determine if initiative is making or has made impact
➢ To provide information for the development of the
initiative or replication of the program
➢ To determine cost effectiveness of program compared
others
➢ Provide information about risk and protective factors
➢ To assess alternative approaches for the prevention or
treatment of health problems

The goal of evaluation is to improve program, thus no evaluation is good unless the
results are used.

5
 Types of evaluation

Harris, J. M. (2017). An introduction to public and community health evaluation. In J. M. Harris, Evaluating
Public and Community Health Programs (pp. 133-170). Hoboken, NJ: John Wiley & Sons Inc.

Evaluation of multipronged interventions – assess both processes and outcomes for

each activity/component, as well as assessing the overall effect of the intervention
Evaluation activities may occur at multiple points on a continuum – from planning the
intervention, through implementation, to assessing effects of populations served

6
 FORMATIVE evaluation

➢ Conducted during the development or delivery of

a program/intervention

➢ Feedback for program improvement

To understand how a program or policy is being implemented or to assess whether an

intervention is likely to have the intended effect
May be done here dynamics, the participants, issues change frequently and program
activities have to adjust to the changing environment
Can be used to pretest educational materials for an intervention or media messages

7
PROCESS evaluation

➢ To determine whether program or plociy is being

implemented appropriately

➢ To determine the extent to which the program is

being implemented as planned

8
 OUTCOME evaluation

➢ To determine the effect of the program or policy

➢ Performed at the end of the intervention or at a

time predetermined by its outcome objectives

The appropriate time for an outcome evaluation is determined by the time line
associated with the program objectives

9
 IMPACT evaluation

➢ To determine extent to which initiatives have

contributed to a population-level effect

10

Outcome and Impact evaluation often used interchangeably

Outcome - used for short, medium and long term changes that can occurs during the
first few years of program/policy implementation
Impact – long term changes in the quality of life at the local, state and national levels
that can be detected by surveillance methods

10
Identifying Stakeholders
Who is…
Affected by the program?
Involved in program operations?
Intended users of evaluation findings?

Who do we need to…

Enhance credibility?
Implement program changes?
Advocate for changes?
Fund, authorize, or expand the program?

there are 3 basic categories of stakeholders:

• those involved in program operations : Staff, managers, and other partners who are directly involved in day-to-
day activities of getting tasks done are those involved in program operations.
• those served or affected by the program: beneficiaries, community groups
• the intended users of the evaluation: may include policy makers, managers, administrators, advocates, and
funders

11
Importance of stakeholders
➢ Framing and selecting the appropriate evaluation
questions
➢ Identifying concepts that are critical to measure
➢ Developing and executing the evaluation plan

12
 Intervention Objectives and Evaluation
➢ Your objectives should be measurable so that they
can be evaluated.
➢ The evaluation should be in line with your
objectives.

13
Internal vs. External Evaluators

INTERNAL

➢Conducted by program employees

➢Knowledge of program
➢Potential bias and influence

14

Internal evaluation – community based or low resourced organizations usually rely on in

house evaluators

14
Internal vs. External Evaluators
EXTERNAL
➢ Conducted by outsiders,
often for a fee
➢ Less visible bias
➢ Outsiders have to gain entry;
have less first-hand
knowledge of the program
15

15
 Evaluation Standards
➢ Utility: Who needs the information and
what information do they need?
➢ Feasibility: How much money, time,
and effort can we put into this?
➢ Propriety: What steps need to be
taken for the evaluation to be ethical?
➢ Accuracy: What design will lead to
accurate information?

One of the primary purposes for using the standards is to help design the best
evaluation for your program or situation. There is no one “right” evaluation. Instead
when designing the evaluation, you make choices that best suit your particular situation
so that the result is the optimal program evaluation for your program or initiative.

The four standards – utility, feasibility, propriety, and accuracy – when applied will result
in the most worthwhile evaluation for your situation. Under each standard on the slide
is a question that can be asked to help design the evaluation. By applying the standards,
you have an evaluation that is useful, feasible, proper and accurate for the program
evaluation that you are conducting at this time.

16
References

1. Harris, J. M. (2017). An introduction to public and

community health evaluation. In J. M. Harris,
Evaluating Public and Community Health Programs
(pp. 133-170). Hoboken, NJ: John Wiley & Sons Inc.