Chapter Eight

Chapter eight
EVALUATION OF EVIDENCE
Tiruneh A. (BSc, MPH)

Purpose of Evaluation of Evidence
To determine if what is observed

is a reflection of the truth?
Observed: Are they true?

Prevalence
Are there
Incidence
alternative
Relative Risk explanations?
Odds Ratio…
Common Problems in observed findings
• Inadequacy of the observed sample
• Inappropriate selection of study subjects
• Inappropriate/unfair data collection

methods
• Comparing unequal groups

Judging observed association
Could it be due to selection or measurement bias?
No
Could it be due to confounding?

No
Could it be a result of chance?
Probably not
Could it be causal?
The role of chance
• we can draw inferences about the experience

of an entire population based on an
evaluation of only a sample.
• Chance may always affect the results
observed simply because of random variation
from sample to sample.
• Sample size is one of the major determinants
of chance.
The role of chance cont…
• the smaller the sample , the more variability

there will be in the estimates and the less likely
the findings will reflect the experience of the
total population.
• the larger the sample on which the estimate is
based, the less variability and the more precise or
reliable the inference made.
• It is important to quantify the degree to which
chance variability may account for the results
observed in any individual study.
• This is done by performing an appropriate test
of statistical significance.
• A measure that is often reported from all tests
of statistical significance is the P value.
• P < 0.05 - statistically significant.
• P> 0.05 – no statistically significant association
( chance can not be excluded as a likely
explanation)
Random Error Vs Bias
Random Error/non-differential Bias / differential
misclassification: misclassification:
• Inaccuracy which is similar in
comparison groups produced • Inaccuracy which is
by biological variations, different in size and
measurement variations, direction in one group
than the other
error during recording etc… (differential error).
• Effect is the same for all • Effect is not the same

groups in the study for groups in the study
Variation
• Clinical determinations are prone to 3 main
sources of variation:
 true biological variation
 temporal variation, and
 measurement errror
The role of bias
• Bias - is any systematic error in the design, conduct, or

analysis of a study that results in a distorted estimate
of what the study is attempting to measure.
• The key word in the understanding of the concept of
bias is “different”.
• If the way in which participants are selected into the
study is different for cases and controls, for example,
and that difference is related to their exposure status,
then the possibility of a bias exists in assessment of
association between the exposure and disease.
The role of bias cont…
• Bias can occur in all types of epidemiologic

studies
• Retrospective studies are more susceptible to
bias than prospective ones
Two main types of bias:
Selection bias
• occur during selection & follow up
Information (Observation) bias
• Occur during data collection
When evaluating a study for the presence of

bias, investigators must:
1. identify its source
2. estimate the magnitude or strength, and
3. assess its direction
Selection Bias
Two broad categories
A)Invitational (who gets invited into the study?)
E.gs
– Berkson's bias
– Healthy worker bias
– Incidence-prevalence bias (missing deaths and
recovered cases)
B) Acceptance (who accepted the invitation?)
E.gs
– Loss to follow-up
– Volunteer/Compliance bias
– Non-response bias
Examples of selection bias cont…
A.1 Berkson’s bias
Case control studies carried out exclusively in

hospital settings are subject to selection bias
attributable to the fact that risks of
hospitalization can combine in patients who
have more than one condition
A.2 Healthy worker bias – any excess risk

associated with a particular occupation will
tend to be underestimated by a comparison
with the general population.
When general population is used as a

comparison group to evaluate the risk of a
particular occupation for diseases and death,
the effect of that occupation may be reduced
because individual working in particular
occupation is more stronger and healthier
than the general population..
Examples of selection bias cont..
A.3. Differential surveillance, Diagnosis, or

Referral
Selection bias can occur in case-control study
as a result of differential surveillance,
diagnosis, or referral of cases that is related to
the exposure
• Example - women who take oral contraceptives (OCs) may

be screened more often for breast cancer than women who
do not take OCs because of the suspected link between oral
contraceptive and breast cancer.
• This would result in breast cancer being diagnosed more
readily in those who are exposed to OCs.
• In turn this would introduce a bias in that exposed cases
may be more likely to come to medical attention and be
included in a study than non-exposed cases.
• False association may be obtained if case control study is
conducted
B.1. Self selection/ Volunteer bias/ Compliance bias

People who accept to participate in a study, or people who
refuse to participate are often quite different from the
general population.
B.2. Non-response bias

• This is due to differences in the characteristics between the
responders and non-responders to the study.
• Non-response reduces the effective sample size, resulting
in loss of precision of the survey estimates.
• Rates of response in many studies may be related to
exposure status.
B. 3. Loss to follow up
• major source of bias in cohort studies
• also a problem in intervention studies
• relates to the necessity of following individuals for a period
of time after exposure to determine the development of
the outcome
• If the proportion of losses to follow-up is large, in the range
of 30 to 40 percent, this would certainly raise serious
doubts about the validity of the study results.
• the more difficult issue for interpretation is that even if the
rate of loss is not that extreme, the probability of loss may
be related to the exposure, to the outcome, or to both.
Ways of minimizing selection bias
1) Population-based studies are preferable
2) the selection of hospitalized controls in a case

control study will increase comparability with
the cases in terms of willingness to
participate, the presence of selective factors
that influenced the subjects’ choice of a
particular hospital, and awareness of
antecedent exposures and events.
Ways of minimizing selection bias cont…
3) One should avoid the inclusions as study

subjects of people who have volunteered on
their own to participate in the study
4) In case-control study, it is useful to select
several different control groups, including if
possible a group selected in the community.
Ways of minimizing selection bias cont…
5) In hospital-based case control study, controls

are usually selected among patients with
diseases other than the disease studied.
-One should ensure that these other diseases
are not related to the exposure & the disease
of interest.
6) keep losses to follow-up to an absolute
minimum.

Information bias /Observation
bias
This refers to bias which arises during the data

collection process, because of mistakes in
categorizing study subjects with respect to
their exposure or disease status
Information bias (Examples)
Investigator bias/ Interviewer bias/ Observer bias
Occurs when investigators collect information differently in different
comparison groups
Recall bias
Occurs as a result of difficulty to recall prior exposures
Social desirability bias
Occurs because subjects are systematically more likely to provide a
socially acceptable response.
Placebo effect
In experimental studies which are not placebo controlled, observed
changes may be ascribed to the positive effect of the subjects belief
that the intervention will be beneficial.
Hawthorn effect – refers to the change in the dependent variable which
may be due to the process of measurement or observation itself
Ways of minimizing information bias
1. Blinding.
Blinding is of greatest importance when the outcome is subjectively
determined.
Three types of blinding

A. Single blind
the study subjects doesn't know to which group they are assigned
B. Double blind
neither the study subjects nor the data collector know the group to
which the subject has been assigned
C. Triple blind
• the study subjects, the data collector and the individual who is doing
the analysis are ignorant of the group to which subjects belong
Placebo
• Placebos are inert treatments intended to have no effect
other than the psychological benefit of offering
treatment.
• Can only be used if there is no accepted treatment for

the condition under study.
• Use of placebo minimizes the bias in the ascertainment

of both subjective disease outcomes and side effects.
• It facilitates that both groups in the study gain equal

attention.
Ways of minimizing information bias cont.
2. same standard procedures, instruments,

questionnaires, interviewing techniques etc
should be used for data collection in both
comparison groups
3. Classification of study subjects according to their
outcome & exposure status should be based on
the most objective & accurate methods available
4. when exposure status is determined by interview,
it should be assessed in several different ways for
both groups, so as to assist all study subjects to
make a thorough attempt at recall
The role of confounding
• Definition - Confounding is a mixing of effects

between an exposure, outcome, and a third
extraneous variable known as a confounder
• Confounder is a variable that can cause or

prevent the outcome of interest, is not an
intermediate variable, and is associated with the
factor under investigation.
The role of confounding cont…
Example of intermediate variable

• Modest alcohol drinking thought to lower the
risk of heart disease by increasing HDL.
• HDL is an intermediate variable (not a
confounder)
Low risk of
Modest Increase heart
alcohol HDL disease
drinking
To bring a confounding effect, that confounding variable
must fulfill each of the following criteria
1) The variable must be associated with the exposure in the
population that produced the cases
i.e the confounder must be more or less common in the
exposed group than the comparison group
2) the variable must be an independent cause or predictor of
the disease
3) Confounder must not be an intermediate link in a causal
pathway between exposure and outcome
4) The confounding variable should be differently distributed
among the comparison groups

Example of confounding effect.
An observed association between
consumption of coffee and increased risk of MI
could be due, at least in part, to the effect of
cigarette smoking, since coffee drinking is
associated with smoking and, independent of
coffee consumption, smoking is a risk factor
for MI.
• Confounding is not an all or none condition

described merely by its presence or absence
• Confounding is a quantitative issue
It is vital to ask the following questions

1) what is the magnitude or extent of
confounding?
• This can be answered by using the following
formula
• Magnitude of confounding =
(RR adjusted – RR crude)/RR adjusted)
2. Direction of confounding
Direction of confounding
• Confounding pulls the observed association
away from the true association
• It can either exaggerate the true association
(positive confounding) or
• Hide the true association (negative
confounding)
• E.g-Observed finding: statistical significant
association between increased level of physical
activity and decreased risk of MI
• Is age a confounder?
• People who exercise heavily are younger than those
who do not exercise
• Independent of exercise, younger individuals have a
lower risk of MI than older people
• Those who exercise could have a lower risk of
MI just simply as a consequence of the greater
proportion of younger individuals in this group
• Age would confound the observed association
between exercise and MI & result in an over
estimate of any inverse relationship
RR
Exercis
e
Exercise +
True
more younger RR=0.5
people in the
exerciser
group
RR=0.3
Overestimation of the effect of exercise on 1

reducing MI
• E.g2 Observed finding: statistical significant association
between increased level of physical activity and decreased
risk of MI
• Is sex a confounder?
• A high level of exercise is likely to be more common in men
• Independent of exercise, men have a greater risk of MI than
women
• An inverse effect of exercise on risk of MI would be
underestimated if differences in gender between exercisers
and non exercisers were not taken in to account
RR
Exercis
e
True
RR=0.5 Exercise +
more men in
the exerciser
group
RR=0.7
Underestimation of the effect of exercise on

reducing MI
1
• Failure to control for positive confounding

results in a more extreme estimate of effect
observed than is actually the case
• This can be in the direction of either an
apparently stronger increased risk or an
apparently more protective, depending on the
direction of true association
• Example of positive confounding is when the

true RR is 2 and the confounded RR is 3.0
• Example of the negative confounding is when
the true RR is 2 and the confounded RR is 1
Null True Observed

value OR = 2 OR= 4
Is this positive or negative confounding?
Observed True
OR=0.2 Null value
OR = 0.5
Null value Observed True

OR= 2 OR=4

Control mechanisms for Confounding
Variables
• In the design:
– Randomization
– Restriction
– Matching
• During analysis:
– Standardization
– Stratified analysis
– Matched analysis
– Multivariate analysis
Confounding and Effect Modification
Effect of Confounding
1.Totally or partially accounts for the apparent

effect
2.Mask an underlying true association
3.Reverse the actual direction of the association

Effect modification (effect
measure modification)
• Effect modification is also known as
heterogeneity of effect
• In effect modification, the association
between the exposure and disease under
study varies by levels of a third factor
• This third factor is called effect modifier
• The term homogeneity of effect is used when
effect modification is absent
Effect modification cont…
Effect measure modification should
be regarded as a causal framework
for gaining insight into a natural
phenomenon
• Example: The modification of the relationship between
cigarette smoking and bronchogenic cancer by asbestos
exposure
• Researchers found that asbestos insulation workers in
general had a 10-fold increased risk of dying from
bronchogenic cancer
• However, they also found that asbestos insulation workers
who smoked had 92 times the risk of dying from
bronchogenic cancer as compared to men who did not
smoke and were not exposed to asbestos
• This tremendous increased risk among individuals with both
exposures reflects the pathological processes whereby the
carcinogens in cigarette smoke and asbestos fibers
exacerbate one another to transform normal cells into
malignant ones.
• Effect modification and confounding are not the
same
• Both involve third variable in addition to the
exposure and disease
• One has nothing to do with the other
• The key difference is that confounding is a
problem that should be eliminated and effect
measure modification is an interesting
occurrence that should be described
• Both can be evaluated by stratified analysis
A particular variable may be
1) a confounder but not an effect modifier
2) an effect modifier but not a confounder
3) both a confounder and an effect modifier
4) neither a confounder nor an effect modifier
Synergy and antagonism
• Are used to describe effect measure
modification
• Synergy (also known positive interaction), is
said to occur when the excess RR among
individuals with both factors is greater than
the sum of the excess RR of each factor
considered alone
Synergy and antagonism cont..
• Antagonism (negative interaction) is said to
occur when the excess RR among individuals
with both factors is less than the sum of the
excess relative risks of each factor considered
alone
• This means that one factor reduces the impact
or even cancels out the effect of the other
factor
Judgment of causality
• Once we found that chance, bias and
confounding are all determined to be
unlikely, then we can conclude that a valid
statistical association exists.
• We should then check other criteria for

judgment of causality
Bradford-Hill criteria
• It is the statement of epidemiological criteria of a
causal association formulated in 1965 by Austin
Bradford Hill
This criteria include:
1. Strength of the Association;

The stronger the association the more likely that it
is causal.
strong --- The more it is far from unity.
---RR> 2 Weak < 2
---RR < 0.5 weak > 0.5
Bradford-Hill criteria cont..
2. Consistency of relationship;
• The same association should be demonstrated by
other studies both with different methods, settings
and different investigators.
• That is why we compare our findings with other

studies.
• Special methods of combining of a number of well

designed studies exist, Meta Analysis
3. Specificity of the association;
Single exposure Single disease
This works more to most living organisms as causes.

4. Temporal relationship
• It is crucial that the cause must precede the
outcome
• The study should show that exposure to be

observed first and outcome next
• This is usually problematic in cross-sectional

and case-control designs.
5. Dose response relationship
• The risk of disease increases with increasing exposure
intensity.
e.g. death due to ischemic heart disease among male
doctors above the age of 45 years followed in
cohort
Death rate Risk Ratio
per 100,000 per-yrs
Non-smokers 7 1.00
1-14 cigarettes/ day 46 6.57
15-24 cigarettes/ day 61 9.71
25+ cigarettes/ day 104 14.86
6. Biological Plausibility:
• Hypothesis should be coherent with what is known about

the disease; both biologically and using laboratory.
• Knowledge about physiology, biology and pathology should

support the cause-effect relationship
• If a finding doesn’t go with what is known, then it is losing

biological plausibility
7. Study design;
It is most important to consider the design.
Type of study Ability to give
evidence
Randomized controlled trial Most Stronger W
E
Cohort study Stronger A
K
Case-control study Moderate N
E
Cross-sectional studies weak S
Ecological studies More weaker S
8. Reversibility
Removal of a possible cause results in a
reduced disease risk
eg. Cessation of cigarette smocking is associated

with reduction in risk of Lung cancer relative
to those who continue
Judging the evidence
There are no completely reliable criteria for determining whether an

association is causal or not.
In judging the different aspects of causation,
• The correct temporal relationship is essential,
• Once this has been found, weight should be given to

– Plausibility,
– Consistency,
– dose-response relationship and
– Strength of the association
• Usually judging for causality is illustrated in
the discussion section of research
• Any finding is judged for such evidence

Chapter Eight

Uploaded by

Document Informationclick to expand document information

Copyright:

Available Formats

Chapter Eight

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter Eight

Uploaded by

Copyright:

Available Formats

Chapter eight

Tiruneh A. (BSc, MPH)

To determine if what is observed

Observed: Are they true?

• Inadequacy of the observed sample

• Inappropriate selection of study subjects

• Inappropriate/unfair data collection

• Comparing unequal groups

Could it be due to selection or measurement bias?

Could it be due to confounding?

Could it be a result of chance?

• we can draw inferences about the experience

• the smaller the sample , the more variability

• Effect is the same for all • Effect is not the same

• Bias - is any systematic error in the design, conduct, or

• Bias can occur in all types of epidemiologic

When evaluating a study for the presence of

A.1 Berkson’s bias

Case control studies carried out exclusively in

A.2 Healthy worker bias – any excess risk

When general population is used as a

A.3. Differential surveillance, Diagnosis, or

• Example - women who take oral contraceptives (OCs) may

B.1. Self selection/ Volunteer bias/ Compliance bias

B.2. Non-response bias

1) Population-based studies are preferable

2) the selection of hospitalized controls in a case

3) One should avoid the inclusions as study

5) In hospital-based case control study, controls

This refers to bias which arises during the data

Three types of blinding

• Can only be used if there is no accepted treatment for

• Use of placebo minimizes the bias in the ascertainment

• It facilitates that both groups in the study gain equal

2. same standard procedures, instruments,

• Definition - Confounding is a mixing of effects

• Confounder is a variable that can cause or

Example of intermediate variable

• Confounding is not an all or none condition

It is vital to ask the following questions

Overestimation of the effect of exercise on 1

Underestimation of the effect of exercise on

• Failure to control for positive confounding

• Example of positive confounding is when the

Null True Observed

Is this positive or negative confounding?

Is this positive or negative confounding?

Null value Observed True

Is this positive or negative confounding?

1.Totally or partially accounts for the apparent

2.Mask an underlying true association

3.Reverse the actual direction of the association

• We should then check other criteria for

1. Strength of the Association;

• That is why we compare our findings with other

• Special methods of combining of a number of well

Single exposure Single disease

This works more to most living organisms as causes.

• The study should show that exposure to be