Monitoring and Evaluation

Evaluation Designs

Population Studies Department

University of Zambia
 Session Objectives
•By the end of the session, students will be
a able to:
1. Name the criteria for inferring causality
2. Understand internal & external validity
3. Know the different types of designs for
Evaluation
4. Identify the strengths and limitations of the
different types of Evaluation designs
5. Selecting an Evaluation design that fits the
purpose of a Program
 Introduction
• To show relationship, Cause or impact,
Evaluators often want to make inferences
about cause and effect in a Program
• We may, for instance, want to:
– Identify the factors that explain why the use of ITNs is
more prevalent among one group compared to
another
– Know why there is low adherence to ART
– Why there is low access to financing for Small Scale
enterprises
 Logic of Causal Inference
• Under what condition is the Cause related to
the Result (dependent variable)
• Under what condition may we infer that a
change in a Population was really caused by
the intervention, and not by something else?
• What are some of the most plausible rival
explanations, and how do we rule them out?
* think about other types of interventions for different
program/projects we are writing our M&E
 Overview
• In this section, we will examine appropriate
and inappropriate criteria for inferring
causality
• We will identify various evaluation designs:
– Experimental
– Quasi-experimental
– Non-Experimental
Criteria for Inferring Causality:
Temporal Relationship

– The cause must precede the effect

Criteria for Inferring Causality:
Plausibility

– An association is plausible, and thus more

likely to be causal, if consistent with other
knowledge
 Criteria for Inferring Causality:
Strength of the Association

– A strong association between possible cause

and effect, as measured by the size of the
relative risk, is more likely to be causal than a
weak association (Testing for significance (Chi
square, Regression)
 Criteria for Inferring Causality:
Consistency
• Several studies giving the same result
– Clearest when variety of study designs are used in
different settings
• The likelihood that all studies are making the same
mistake is minimized
– Lack of consistency does not exclude a causal
association
• Different exposure levels and other conditions may
reduce the impact of the causal factor in certain studies
 Criteria for Inferring Causality:
Dose-response Relationship
• Occurs when changes in the level of a
possible cause are associated with changes
in the prevalence or incidence of the effect
– The prevalence of hearing loss increases with
noise level and exposure time
Criteria for Inferring Causality:
Reversibility
• When the removal of a possible cause results in a
reduced disease risk, the likelihood of the
association being causal is strengthened
– Cessation of cigarette smoking is associated with a reduction
in the risk of lung cancer relative to that in people who
continue to smoke
• However, when the cause leads to rapid irreversible
changes whether or not there is continued
exposure (HIV infection), then reversibility cannot
be a condition for causality
 Reliability
• Implies that the same data would have
been collected each time over repeated
tests/ observations.
• Would a particular technique (or survey
question) yield the same result each time?

Reliability does not ensure Accuracy

Internal & External Validity
Internal & External Validity
• When considering cause and effect, two
forms of validity become relevant:
– Internal validity
– External Validity
 Internal Validity
• The degree to which the results are
attributable to the independent variable
and not some other rival explanation

• The confidence that the results of a study

accurately depict whether one variable is or
is not a cause of another
 Internal Validity
• Validity is the extent to which a test
measures what it is supposed to measure.
• The question of validity is raised in the
context of the three points;
a. The form of the test,
b. The purpose of the test and,
c. The population for whom it is intended
 Threats to Internal Validity
• History
• Maturation or the passage of time
• Testing & Instrumentation
• Selection bias/differential selection of
participants
• Loss to follow-up/Mortality
• Diffusion or imitation of treatments
 Why is Internal Validity
Important
• We often conduct research in order to
determine cause-and-effect relationships.

• Can we conclude that changes in the

independent variable caused the observed
changes in the dependent variable?

• Is the evidence for such a conclusion good

or poor?
 Cont..
• If a study shows a high degree of internal
validity, then we can conclude we have
strong evidence of causality.

• If a study has low internal validity, then we

must conclude that we have little or no
evidence of causality
 External Validity

• External validity refers to how a study's

results can be generalized to a larger
population. In this case, validity is
determined in part by whether a study's
outcomes can be replicated in and across
other samples, times and settings. A
study with high external validity can
therefore be repeated in multiple contexts
with similar outcomes
 Threats to External Validity
• Pre-test treatment interaction
• Multiple treatment interference
• Selection treatment interaction
• Specificity of variables
– Participants
– Operational definition of the treatment
– Operational definition of the dependent variable
– Specific times
– Specific circumstances
• Treatment diffusion and inconsistencies
 Bias in Research
• Bias definition: “Error which produces results or
conclusions that differ systematically from the
truth”
Bias Validity

• Can occur at any point during a study – study

design, sampling, data collection, analysis and
interpretation of results
 Confounding
• Confounding is an important source of bias
– The exposure (E) is associated with a third factor, the
confounder (CF), which is also linked to the disease (D)
– It can distort a true association/Relationship or create
a spurious association

CF

E ?? D
 Hierarchy of Evidence
Experimental Strongest
• Systematic review
• Experimental study (RCT) Less bias
Quasi Experimental
• Cohort study
• Case-control study
Non-Experimental
• Cross-sectional study More bias
• Case series
Weakest
 Picking on a Research Design

1. What is your research question?

2. What do you want to measure (Dependent Variable)?
3. What is your target population?
4. Do you have a sampling frame? What shape is it in?
5. What do you know about this population?
6. How do you intend to use the results?
7. What level of analysis are you going to perform
8. What type of inference do you want to draw?
9. How much are you willing to pay?
10. What does literature say about question of interest
 Types of Research Design

Design
Experimental Strongest for demonstrating
causality, most expensive
Quasi- Weaker for demonstrating causality,
experimental less expensive

Non- Weakest for demonstrating causality,

experimental least expensive (explorative studies)
Experimental Design
 Experimental Design
• experimental design is a research design
that eliminates all factors that influence
outcome except for the cause being studied
(independent variable).
• All other factors are controlled by
randomization, investigator-controlled
manipulation of the independent variable,
and control of the study situation by the
investigator
The Basic Experimental Principle

• The intervention is the only difference

between two groups. The two groups should
be equivalent except one receives an
intervention
• Experimental design is about IF X THEN Y
If the intervention/effect/cause is given, then
the outcome occurs
• Experimental design is achieved by random
assignment
Types of Experimental Design
 A. Pretest-posttest Experimental Design

• Experimental designs attempt to provide

maximum control for threats to internal
validity
• They do so by giving the researchers greater
ability to manipulate and isolate the
independent variable
– (not always possible in practice)
 Pretest-posttest Experimental Design
Essential components:
1. Identify the study population and determine the appropriate
sample size for experimental and control groups
2. Randomly assign individuals to experimental and control
groups
3. Pre-test everyone with a standardized instrument
4. Introduce the independent variable (intervention) to the
experimental group while withholding it from the control
group
5. Post-test both groups with the same instrument and under
the same conditions as the pretest
6. Compare the amount of change in dependent variable for
both experimental and control groups
 Pretest-posttest Control Group Design

Intervention Pre- Post-

group test Program test

Randomization

Control Pre- Post-

group test test

Time
 Factors that May Distort Conclusions

Dropout
Instrumentation effects

Testing effects
If you think that taking the pretest might
influence the treatment effects, or if it might
bias the post-test responses, you might want to
opt for the posttest-only control group design
Contamination
Posttest-only Experimental
Design
Post Test Only Experimental Design

• Initial group equivalence

• Differences between the experimental
and control groups at posttest are
assumed to reflect the causal impact of
the independent variable
 Post Test Only Experimental Design

Intervention Post-
Program
group test

Randomization

Control Post-
group test

Time
 Post-test Only, what to consider?

Advantages Disadvantages
 Cheaper  Cannot assess whether the
 Useful when pre-test can program is going to people
interfere with program for whom it was intended
effects  Cannot check comparability
 Randomization ensures of groups
equivalent experimental and  Cannot know how much
control groups change actually occurred

However,
However,aapre-test
pre-testpost-test
post-testdesign
designalways
alwayspreferred
preferred
 Group Discussion
In which situations might experimental
design not be possible?
 Possible Responses
• Randomization needed to assured
• Ethics
o Solution: Use alternative program rather than
no program
o Known efficacy of intervention
• Political factors
• Scale-up
o Solution: Start out on small scale and use
delayed program strategy
Quasi-Experimental Design
 Quasi-experimental designs
• Can be used when random assignment is not
possible
• In this type of experiment subjects are not
assigned randomly to control and
experimental groups as in the experimental ds
• Less internal validity than “true” experiments
• Still provide a moderate amount of support for
causal inferences
Principles of the Quasi-experimental
Design
(Pre- and Post-test with comparison group but not randomized)

Intervention Pre- Post-

Program
group test test

Comparison Pre- Post-

group test test

Time

Keep
Keep in
in mind
mind selection
selection effects:
effects: these
these occur
occur when
when people
people selected
selected for
for aa
comparison
comparison group
group differ
differ from
from the
the experimental
experimental group
group
Quasi-experimental Statistical Methods
o Difference-in-Difference analysis: method
involves comparing changes before and
after the program for individuals in the
program and control groups
o Regression analysis: Attempts to address
the problem of confounding by controlling
for difference at baseline
 Summary of Quasi-Experimental Design

Advantage Disadvantage
 Provides the assurance that  Can demand more time and
outcomes are actually the resources
results of the program  Require access to at least two
 Allows you to accurately similar groups
assess how much of an effect
the program has
Non-Experimental Design
Pre-test Post-test Non-experimental design

Pre-test Post-test (No comparison group )

Intervention Pre- Post-

Program
group test test

Time
 Cross Sectional Design
• cross-sectional design a research strategy
in which one or more group(s) of subjects
are studied at one given point in time.
• This is a snapshot of a population at a
certain time, allowing conclusion to drawn
about a population.
• E.g Prevalence of Cervical cancer in a
population in 2017
Cross Sectional Study (D)
 Case Study/Series

• Based on reports of a series of

cases of a specific condition, or a
series of treated cases, with no
specifically allocated control
group.
 Merits of Pre-test Post-test Non-experimental
design

Advantage Disadvantage
 Relatively simple to  Cannot account for non-
implement program influences on
outcomes
 Controls for participants'  Causal attribution not possible
prior
 Cannot detect small but
knowledge/attitudes/skills
intentions important changes
 If self-reporting is used rather
than objective measures,
posttest scores may be lower
than pretest scores
 Time Series Design
(No comparison group )

Intervention Pretest Pretest Posttest Posttest

1 2
Program 1 2
group

Time
 Merits of Time Series Design

Advantage Disadvantage
 Enables detection of • Problem of confounding
whether program effects are
long-term or short-term • Changes in instruments
 Series of tests before during the series of
intervention can eliminate measurements
need of control group
• Loss or change of cases
 Series of tests before
program can be used to • Changes in group
project results which would composition
be expected
 Can be used if you have only
one site to conduct your
evaluation
 Strengthening non-experimental designs
• Since there is no control group confounding
can be a problem
• By constructing a plausibility argument and
controlling for contextual and confounding
factors, non-experimental designs can be
strengthened
 How to construct a
Plausibility Argument
• Describe trends in
• Intervention coverage
• Intermediary outcomes
• Impact outcomes
• Contextual factors
• Link these trends
– Temporal, spatial, age-pattern, “dose-response”
associations
 Plausibility Argument for
Impact of Malaria Control
Key Variables

• ITN Ownership • Malaria Parasite All cause under-five

• ITN use prevalence mortality (5q0)
• Treatment • Fever

Decreased
Increase in effective Decreased malaria-
intervention coverage morbidity associated
mortality
Contextual factors

• Socioeconomic
• Health intervention
• Climatic factor • Education
• Health care
• Rainfall • Fertility risk
utilization
• Temperature • Housing condition
• ANC, Vit A, PMTC
• Nutrition
 Summary of Different Study Designs

True experimental Quasi-experimental Non-experimental

Partial coverage/ Partial coverage/ new Full coverage
new programs programs programs

Control group Comparison group

--
Strongest design Weaker than
experimental design Weakest design

Most expensive Less expensive

Least expensive

More robust Less robust

 Summary of Different Study Designs

• Different designs vary in their capacity to

produce information that allows for the linking of
program outcomes to program activities

• The more confident you want to be about

making these connections, the more rigorous the
design and costly the evaluation
 Group work
• Please get into your project groups and
select an Research design for your
program (Experimental, Quasi-
Experimental, Non-Experimental)
• Explain why you chose this design
• Discuss any strengths and weaknesses of
this design as they relate to your
Research Topic
General Questions to be answered
by Each Group
• What study design will you use?
• What are the strengths and limitations of your
evaluation design?
• How would you know if you’re your topic will be
answered?
Qualitative Research Designs
 Qualitative Study
• Commonly referred to as a systematic
subjective approach used to describe life
experiences and give them meaning
• Aims at gaining insight; explore the depth,
richness, and complexity inherent in the
phenomenon.
 Characteristics of Qualitative
Studies
• Soft science
• Focus: complex & broad
• Holistic
• Subjective
• Dialectic, inductive reasoning
• Basis of knowing: meaning & discovery
• Develops theory
• Shared interpretation
• Communication & observation
• Basic element of analysis: words
• Individual interpretation
• Uniqueness
 Qualitative Research Designs
• Phenomenology
• Grounded theory
• Ethnography
• Historical
• Case study
 Phenomenology
• Examines uniqueness of individual's lived
situations
• Each person has own reality; reality is subjective
• Seek persons who understand & are willing to
express inner feelings & experiences
• Describe experiences of phenomenon (event)
– Method Used
• Write experiences of phenomenon
• Direct observation
• Audio or videotape
 Grounded theory
• Used in discovering what problems exist in a
social scene &how persons handle them
• Involves formulation, testing, &
redevelopment of propositions until a theory
is developed
Method Used
• interview, observation, record review, or
combination
 Ethnography
• Aims at describing a culture's characteristics

Method Used
• Gain entrance to culture; immerse self in
culture; acquire informants; gather data
through direct observation & interaction with
subjects
 Historical
• Describe and examine events of the past to
understand the present and anticipate
potential future effects
Method Used
• Formulate idea - select topic after reading related literature
• Develop research questions
• Develop an inventory of sources - archives, private libraries,
papers
• Clarify validity & reliability of data - primary sources,
authenticity, biases
• Develop research outline to organize investigative process
• Collect data
 Case Study
• Describe in-depth the experience of one
person, family, group, community, or
institution

Method Used
• Direct observation and interaction with
subject
 Mixed Methods
• These are research designs that uses both
qualitative and Quantitative methods to
solicit information (Predetermined and
emerging methods)
• Both open and closed-ended questions
• Takes advantage of using multiple ways to
explore a research problem
• Thus, uses multiple forms of data drawing
on possibilities statistical and text analysis
 Why use Mixed Methods
• Explain and interpret
• Explore a phenomenon (event)
• Develop or test a new instrument
• Serves as a theoretical perspective
• Complement the strengths of a single design
• address questions at different levels
• Addresses theoretical perspective at
different levels
 Strengths and Weaknesses of
Mixed Methods
Strengths Weaknesses
• Easy to describe and • Time consuming
report • Can generate unequal
• Useful when evidence
unexpected results • Difficult to decide
arise when to proceed
• Helpful in designing • Difficult in resolving
and validating tools discrepancies between
different data types
METHODOLOGY
Key Features
 Methodology
• A. Using Secondary data set
• B. Using primary data sources (Collection
own data)
 Secondary Data Sets
a. Study Design
b. Source of Data
c. Description of the Data set(s)
d. Sample size description
e. Operationalization of Variables
a. Independent Variables
b. Dependent Variable
f. Data analysis
g. Ethical issues
h. Limitations
 Primary Data (Collecting own
Data)
a. Study type (Design)
b. Focus Population/Sample
c. Location
d. Sampling Techniques
e. Sampling Procedures
f. Materials (Tools: Questionnaires, Interview guides,
FGDs schedule etc.)
g. Variables
a. Dependent Variable
b. Independent Variable
h. Analysis (Qualitative, Quantitative)
i. Limitations
Next Class

Sampling Techniques