Public Administration and Policy
PAD634 Judgment and Decision Making Behavior
Coherence and correspondence
Thomas R. Stewart, Ph.D.
Center for Policy Research
Rockefeller College of Public Affairs and Policy
University at Albany
State University of New York
[email protected] Copyright © Thomas R. Stewart
Coherence research
Coherence research measures the quality
of judgment against the standards of
logic, mathematics, and probability theory.
Coherence theory argues that decisions
under uncertainty should be coherent,
with respect to the principles of
probability theory.
coherence-correspondence.ppt 2
Coherence
Representativeness problem
There are two programs in a high school. Boys
are a majority (65%) in program A, and a
minority (45%) in Program B. There is an equal
number of classes in each of the two programs.
You enter a class at random, and observe that
55% of the students are boys. What is your best
guess--does the class belong to program A or
program B? (Kahneman and Tversky, 1972)
coherence-correspondence.ppt 3
Coherence
Representativeness problem
Most subjects (67 out of 89) picked program A
because the class had more boys, and so did
program A. This is an example of an important
heuristic: “representativeness.” We judge an
event to be a member of a set of events if its
characteristics are similar to our belief about
members of that set. This is a good rule of
thumb in a lot of situations, but not this one.
coherence-correspondence.ppt 4
Coherence
Representativeness problem
Some people might feel that the answer
could be “either program” because 55%
is midway between 45% and 65%.
They would have noticed a quantitative
relationship, and made a reasonable
guess.
But they would be wrong.
coherence-correspondence.ppt 5
Coherence
Representativeness problem
The correct answer is Program B.
The variance of a binomial will be larger for
p = .45 than for p = .65. Therefore, there
will be more variability in the number of
boys in Program B’s classes, and it is
more likely that a class in Program B will
have as many as 55% boys.
coherence-correspondence.ppt 6
Coherence
Representativeness problem
Why does it matter that students (in this case
Israeli high school students) fail such a difficult
question? Unfortunately, they also fail easier
questions. Furthermore, its not only students
who make these mistakes. These kinds of
errors have been found in studies of scientists,
statisticians, physicians, and other
professionals.
coherence-correspondence.ppt 7
Coherence
Mammography illustration
David Eddy (1982) asked us to imagine that a physician
has a patient with a breast mass that he thinks is
probably benign. Lets interpret “probably” as a
probability of .99. The physician is 99% sure, but not
completely sure, that the mass is benign. This is his
“prior probability.” The physician orders a
mammogram and receives a report that the
radiologist thinks the mass is malignant. What
should the physician do?
coherence-correspondence.ppt 8
Coherence
Mammography illustration
If he read the literature available in 1982,
he would find statements like:
• “The accuracy of mammography is
approximately 90 percent”
• “A positive report of carcinoma is highly
accurate.”
• “The accuracy of mammography in correctly
diagnosing malignant lesions averages 80 to
85 percent.”
coherence-correspondence.ppt 9
Coherence
Mammography illustration
Note that this is NOT a decision table.
If he wanted more specific data, he could find these.
Mammograpy outcomes (from Eddy, 1982)
If malignant lesion If benign lesion (no
Results of X ray (cancer) cancer)
P(Positive test) = .792 .096
P(Negative test) = .208 .904
Sensitivity = .792 Specificity = .904
coherence-correspondence.ppt 10
Coherence
Mammography illustration
The physician’s subsequent action for this
patient might be influenced by his
judgment about the likelihood that the
woman has a malignant lesion. What
would be your judgment about that
likelihood based on these data?
coherence-correspondence.ppt 11
Coherence
Mammography illustration
• Eddy sampled 100 physicians and
found that most of them thought the
probability would be around 75 percent.
• We can calculate that likelihood using
the data given and Bayes theorem:
coherence-correspondence.ppt 12
Coherence
Bayes Theorem
P(pos|ca)P(ca)
P(ca|pos)
P(pos|ca)P(ca) P(pos|benign)P(benign)
ca = malignant lesion (cancer) benign = benign lesion (no cancer) pos = positive test
(0.792)(0.01)
P(ca| pos) 0.077
(0.792)(0.01) (0.096)(0.99)
coherence-correspondence.ppt 13
Coherence
Mammography illustration
• So, the actual probability is about .08. The
physicians’ judgments were off by almost a
factor of 10!
• Does this matter?
– It only matters if the previous statement is true,
that is, “the physician’s subsequent action might
be influenced by his judgment about the likelihood
that the woman has a malignant lesion.” In fact,
this is a basic tenet of coherence research.
coherence-correspondence.ppt 14
Coherence
Fundamental tenet of coherence
research
"Probabilistic thinking is important if people are to
understand and cope successfully with real-world
uncertainty."
Example:
"Accurate estimations of probability in diagnosis and prognosis are
important for physicians, because these probabilities influence their
diagnostic and therapeutic decisions. Objective probabilities, based on
epidemiological research, are often not available. In these instances,
physicians must rely on their own judgment and apply their probability
estimates to individual patients. Research has shown that the accuracy of
probability estimates of both experts, such as physicians and of lay people
in general is often inadequate...." Timmermans et al. (1996, pp. 107-108)
coherence-correspondence.ppt 15
Coherence
Fundamental tenet of coherence
research
"Probabilistic thinking is important if people are to
understand and cope successfully with real-world
uncertainty."
Example:
"Accurate estimations of probability in diagnosis and prognosis are
important• for
Two elements
physicians, of thisthese
because quote:
probabilities influence their
• Probabilitydecisions.
diagnostic and therapeutic Objective
judgments are probabilities,
important because they based on
epidemiological research,
influenceare often not
decisions available.
made In these instances,
under uncertainty.
physicians must •rely on their
People own
are not judgment
very and apply
good at making their probability
probability
estimates to individual patients. Research has shown that the accuracy of
judgments.
probability estimates of both experts, such as physicians and of lay people
in general is often inadequate...." Timmermans et al. (1996, pp. 107-108)
coherence-correspondence.ppt 16
Correspondence
Correspondence research
Correspondence research measures the
quality of judgment against the standards
of empirical accuracy. Correspondence
theory argues that decisions under
uncertainty should result in the least
number of errors possible, within the
limits imposed by irreducible uncertainty.
coherence-correspondence.ppt 17
Correspondence
Fundamental tenet of correspondence
research
"Human competence in making judgments
and decisions under uncertainty is
impressive. Sometimes performance is
not. Why? Because sometimes task
conditions degrade the accuracy of
judgment."
Hammond, K. R. (1996). Human Judgment and Social Policy:
Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice.
New York, Oxford University Press (p. 282).
coherence-correspondence.ppt 18
Coherence and correspondence
theories of competence
Coherence theory of competence
Uncertainty irrationality error
Correspondence theory of competence
Uncertainty inaccuracy error
What is the relation between coherence and
correspondence?
coherence-correspondence.ppt 19
Comparison
Comparison of Coherence and
Correspondence Theories
•Sources of error
•Sources of disagreement
•Methods for improving judgment
•Value judgments
•Non-repetitive decisions
•Sources of competence
coherence-correspondence.ppt 20
Comparison
Sources of error
•Coherence
"Irrationality" with respect to a normative system
•Correspondence
•Multiple fallible indicators:
Unreliability in information acquisition
Unreliability in information processing
Match between environment and judge
Fidelity of information system
Environmental uncertainty
Bias
coherence-correspondence.ppt 21
Comparison
Sources of disagreement
•Coherence
Use of different heuristics
•Correspondence
Same as sources of error:
Unreliability in information acquisition
Unreliability in information processing
Match between environment and judge
Fidelity of information system
Environmental uncertainty
Bias
coherence-correspondence.ppt 22
Comparison
Methods for improving judgment
•Coherence
Decomposition and mechanical recomposition
Training in probability and statistics
Formal elicitation of subjective probabilities
•Correspondence
Environmental changes, training, and cognitive aids to
address:
Unreliability in information acquisition
Unreliability in information processing
Match between environment and judge
Bias
coherence-correspondence.ppt 23
Comparison
Value judgments
•Coherence
Decision theory assumes values can be
elicited.
•Correspondence
Has little to say. Correspondence not
relevant.
coherence-correspondence.ppt 24
Comparison
Non-repetitive decisions
•Coherence
Decision theory provides prescriptive method.
•Correspondence
Not clear how to apply correspondence theory
when decisions are not repetitive.
coherence-correspondence.ppt 25
Comparison
Sources of competence
•Coherence
Must be taught, usually in a formal setting, like
a school.
•Correspondence
Use of multiple fallible indicators is innate.
coherence-correspondence.ppt 26
Integration of coherence and
correspondence theories
•Learn/teach identification of tasks
demanding coherence.
•Learn/teach identification of tasks
demanding correspondence.
coherence-correspondence.ppt 27
