Background

Validity and Reliability of • Health/Social sciences extensively use measurement

Measurements • Measurements vary across different disciplines
• They include:
– Clinical assessments (history and physical examination)
– Physical measurements of (say weight, height, length)
– Laboratory assessment
– Social Assessments
Dr. Mayega Roy William – Questionnaires to assess social attributes
– Frequency of diseases, conditions or events in populations
• Every procedure we conduct, for the purpose of coming
to a conclusion, is a test.
• However, procedures differ in their level of accuracy and
certainty
8/31/2012 1 8/31/2012 2

Measurement is the spice of life:

Mirror, Mirror! Who is the fairest of them Ideal and Real Situations
all? • You may ask: Why not use the ideal test?
• Ideal tests are often hard to conduct on a Public
Health basis
• They may be invasive, expensive or highly
technical
• They therefore cannot be used on a mass scale
• They are only used in specialised centres like
Hospitals, research institutes etc for a selected
range of cases
• At a population level, we need simple tests
8/31/2012 3
•8/31/2012
However, the tests ought to be valid 4

Components of Validity and

Why validate tests?
Reliability
• Because we cannot use ideal tests • Validity:
• Whenever you make a measurement, – Sensitivity and Specificity
think about how closely it reflects the truth – Positive and Negative Predictive Value
• Was the result you got the real truth?
• Could there be a better test? • Reliability:
– Precision
• Could it change if the measurement was
repeated several times? – Accuracy

• It is common to record variation when a

measurement is repeated several times 5
8/31/2012 8/31/2012 6

1
 Validity
Validity
• The extent to which a test measures
correctly what it is intended to measure • It has two components:
– 1. Sensitivity: The ability of a measurement
to identify correctly those who have a
• The ability of a test to correctly particular condition
differentiate between who has a particular
condition and who does not – 2. Specificity: The ability of a measurement
to identify correctly those who do not have a
condition

8/31/2012 7 8/31/2012 8

Determining validity 2X2 Table

• We use another test considered the “Gold Disease
standard” Test With D’se Without
– This test cannot be used routinely because of D’se
its invasiveness or expensiveness +Ve a (TP) b (FP) (a+b) or
(TP + FP)
– If the outcomes are dichotomous we can - Ve c (FN) d (TN) (c+d) or
represent the counts in a 2X2 epidemiological (FN + TN)
table
(a+c) or (b+d) or (a+b+c+d)
(TP+ FN) (FP + TN)
8/31/2012 9 8/31/2012 10

Interpreting the 2X2 Table I Interpreting the 2X2 Table II

• Sensitivity = TP or a • Sensitivity: Probability of testing positive
TP + FN (a+c) when one has the disease (or proportion of
persons who are actually ill, that the test
correctly detects)

• Specificity = TN or d
• Specificity: Probability of testing negative
FP + TN (b+d) when one has no disease (or proportion of
persons who are not actually ill that the test
correctly identifies as negative)
8/31/2012 11 8/31/2012 12

2
 Example Example
• Imagine a population of 500 people of Disease
whom 100 actually have a disease and Test With D’se Without D’se
400 do not have the disease. Imagine +Ve 90 110 200
also that all the 500 people are subjected - Ve 10 290 300
to a screening test that is capable of 100 400 500
identifying positive cases, with the
following results. • From this table therefore:
– Sensitivity = 90/100 = 90%
– Specificity = 290/400 = 72.5%
8/31/2012 13 8/31/2012 14

Dealing with Continuous

Illustration: Systolic BP
Variables
• So far, we have been dealing with discrete • Individuals differ Distribution of Systolic BP in a Population X
dichotomous variables. biologically and
as such, 140
• When dealing with continuous variables

Number of persons
120
continuous 100
(e.g. Blood sugar or Blood pressure), the variables are 80
Series1
challenge is even greater distributed as 60
40
• We have to establish cut – offs ranges 20
0
• Decision is made basing on cut-offs 80 90 100 110 120 130 140 150 160
• Normal Bystolic BP
Distribution
8/31/2012 15 8/31/2012 16

Dealing with multi-level variables Consequences of False tests

• This requires advanced statistical False Tests: Do unto others what you would like to be done on
you
procedures False Positivity- False Negativity
• Requires Factor Analysis Emotional/Psychological- -A potentially serious condition
Burdening the health system-It is may be missed
• It is outside the scope of this session difficult to undo one’s thinking is
the true situation is discovered

-Think about telling some one that -Think about missing early cancer
they are HIV positive when they of the breast or cervix, which
are not would be cured if detected at the
-Think about over treatment of earliest histological staging
STDs -Think about the Blood Bank
8/31/2012 17 -The Kiboga Dilemma
8/31/2012 -Think about organ donation 18

3
 Improving Validity of What happens in sequential
Measurements testing?
Test 1: if: Sensitivity of this test is 70%
• Sensitivity: Specificity of this test is 80%
– Two stage testing – simultaneous Total Population: 10,000
• Prevalence of diabetes in population (Known): 5% therefore
– Two stage testing – test with higher sensitivity those with diabetes are 500
• When we apply this test, with the above sensitivity and specificity,
• Specificity: we are likely to get the following :

– Two stage testing – sequential Disease

– Two stage testing – test with higher specificity Test With D’se Without D’se
+Ve 350 1900 2250
- Ve 150 7600 7750
8/31/2012 19 8/31/2012 500 9500 10,000 20

What happens in sequential What happens in sequential

testing? testing?
• In sequential testing, we subject the only Test 2: if: Sensitivity of this test is 90%
Specificity of the test is 90%
those who turn out positive to a second Total Population: Positives retested = 2550
• Prevalence: In the first test, those with actual diabetes were 500; in
test the second test however, we exclude those who tested negative.
• Those that turn negative with the second Therefore, those with actual diabetes are 350, meaning that the rest
do not have (1900). We then fill in the newly computed figures.
test are then reported as positive
• Those that turn positive with the second Disease
test are reported as positive Test With D’se Without D’se
• Those that are indeterminate are +Ve 315 190 505
subjected to a “tie-breaker test” - Ve 35 1710 1745
8/31/2012 21 8/31/2012 350 1900 2250 22

What happens in sequential What happens in simultaneous

testing? testing?
Therefore: • In simultaneous testing, the same sample
• Net sensitivity = 315/500 X 100 = 63% is subjected to different tests at the same
– Sequential testing results into overall time
reduction in net sensitivity
• For one to be negative, they have to pass
all the tests as negative (overall loss in
• Net specificity = 7600(from test 1) + 1710 specificity).
(from test 2)/ 9500 X 100 = 98%
– Sequential testing results into overall increase • One is positive if they fail any of the
in net specificity several tests (overall gain in sensitivity)
8/31/2012 23 8/31/2012 24

4
 Guideline for decision Predictive value of tests
• Both sensitivity and specificity are desirable In Public Interventions
• However, it is not always possible to achieve • Interested is in screening populations to find
maximum sensitivity and specificity what proportion is affected by a disease/situation
– e.g. we want to correctly identify as many people with
– Depends on the use to which the test is STDs for early treatment
– For Blood Bank – high sensitivity – We are more interested in sensitivity and specificity
– For VCT – high specificity In Specialised practice
– For population based studies, we need both in • The physician is more interested in tying up their
equal measure findings to make a diagnosis
– More interested in answering the question:
– A test with low validity should be trashed – If the answer to a question is positive, is this person
actually telling the truth?
8/31/2012 25 8/31/2012 26

Illustration Definition
• The PV of a measurement is the probability that
- If we are interested in a community development those it shows as having a particular condition
project for the most poor (Bonna-bagaggawale) actually have it.
- And we simply ask a person “are you very poor” • It has two components:
– everyone will reply in the positive! – 1. Positive predictive value: The proportion of those
- The Predictive Value of such a method may be who test positive that in reality has the disease
poor (probability that a person has disease, given that he/she
- On the other hand, if we used the nature of tests positive)
house they stay in (Mud, Bricks etc), we may get
better information – 2. Negative predictive value: The proportion of those
- Some cultures believe that every body has who test negative that in reality does not have the
“Kabootongo” or “Syphilis” disease (probability that a person has no disease, given
8/31/2012 27 that he/she tests negative)
8/31/2012 28

2X2 Table Interpreting the 2X2 Table III

Disease • PPV = TP or a
Test With D’se Without TP + FP (a+b)
D’se
+Ve a (TP) b (FP) (a+b) or
• NPV) = TN or d
(TP + FP)
FN + TN (c+d)
- Ve c (FN) d (TN) (c+d) or
(FN + TN)
(a+c) or (b+d) or (a+b+c+d)
(TP+ FN) (FP + TN)
8/31/2012 29 8/31/2012 30

5
 Relationship bet’n predictive Example – PPV and Prevalence
value and Prevalence
• Scenario 1: Population with low prevalence of
• Unlike sensitivity and specificity that apply malaria
– Population 1: 10,000pple
directly to the inherent properties of a test, – Test used: Sensitivity: 90%
the predictive value of a test is affected by Specificity: 90%
– Disease: Malaria
two factors: – Prevalence: 1% therefore expected diseased are: 100

– Prevalence of the Disease in a population

Disease
– Specificity of the test
Test With D’se Without D’se
• We shall only deal with prevalence
+Ve 90 990 1080
- Ve 10 8910 8920
8/31/2012 31 8/31/2012 100 9900 10,000 32

Example – PPV and Prevalence Example (Ctd)

• Scenario 2: Population with higher prevalence of malaria
• Determine Predictive Values (PPV and – Population 1: 10,000pple
– Test used: Sensitivity: 90%
NPV) Specificity: 90%
• What do we obtain? – Disease: Malaria
• Prevalence: 10% therefore expected diseased are: 1000
• Therefore: • Determine the PPV and NPV

– PPV = 90/1080 X 100 = 8.3% NOTE! Disease

– NPV = 8910/8920 X 100 = 99.8% Test With D’se Without D’se
+Ve 900 900 1800
- Ve 100 8100 8200
8/31/2012 33 8/31/2012 1000 9000 10000 34

Example (Ctd) Reliability of measurements

• Part 1: • Synonymous with Repeatability:
– PPV = 90/1080 X 100 = 8.3% NOTE! – Related to consistency of a measurement technique
– NPV = 8910/8920 X 100 = 99.8% – When the same measurement is repeated many
• Part 2: times and the same result is obtained all the time,
– PPV = 900/1710 X 100 = 50%: NOTE! then the test is said to be reliable
– NPV = 8190/8290 X 100 = 98.8% – Eg. If a weighing scale measures the same object
• Deduction: many times and it records the same weight all the
time then that scale is reliable
• PV increases with prevalence
– If each result is different from the others then the
• A simple test for poverty will apply in populations scale has got low reliability
with high risk of poverty but it will not apply in
– It is important, however, to note that reliability alone
populations with low risk
8/31/2012 35 8/31/2012does not necessarily indicate validity 36

6
 Precision and Validity Illustration
• Precision: How close are the repeated
measurements?

• Accuracy: How correct are the

measurements? (Validity)

8/31/2012 37 8/31/2012 38

Factors that affect repeatability Percentage Agreement

• Intra-subject Variation
– Basal temperature • Compare the Obs. 1
– BP proportion of
– Nocturnal and Ambulatory length readings in which +Ve -Ve
both observers
agree Obs. +Ve a b
• Inter Observer variation • Against the total 2
– Diagnostic acumen number of readings: -Ve c d
– Characteristics of observers • Percentage
agreement is given
• Changes in Validity
– Changing case definitions
by: ad
– Changing Natural History
8/31/2012 39 abcd 40

End!

8/31/2012 41