See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/222511520

Introduction to ROC analysis

Article  in  Pattern Recognition Letters · June 2006

DOI: 10.1016/j.patrec.2005.10.010 · Source: DBLP

CITATIONS READS

10,951 11,884

1 author:

Tom Fawcett
Silicon Valley Data Science
46 PUBLICATIONS   19,850 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Feature generation View project

ROC analysis View project

All content following this page was uploaded by Tom Fawcett on 06 April 2018.

The user has requested enhancement of the downloaded file.

 Pattern Recognition Letters 27 (2006) 861–874
www.elsevier.com/locate/patrec

An introduction to ROC analysis

Tom Fawcett
Institute for the Study of Learning and Expertise, 2164 Staunton Court, Palo Alto, CA 94306, USA

Available online 19 December 2005

Abstract

Receiver operating characteristics (ROC) graphs are useful for organizing classifiers and visualizing their performance. ROC graphs
are commonly used in medical decision making, and in recent years have been used increasingly in machine learning and data mining
research. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice.
The purpose of this article is to serve as an introduction to ROC graphs and as a guide for using them in research.
 2005 Elsevier B.V. All rights reserved.

Keywords: ROC analysis; Classifier evaluation; Evaluation metrics

1. Introduction domains with skewed class distribution and unequal clas-

sification error costs. These characteristics have become
A receiver operating characteristics (ROC) graph is a increasingly important as research continues into the areas
technique for visualizing, organizing and selecting classifi- of cost-sensitive learning and learning in the presence of
ers based on their performance. ROC graphs have long unbalanced classes.
been used in signal detection theory to depict the tradeoff ROC graphs are conceptually simple, but there are some
between hit rates and false alarm rates of classifiers (Egan, non-obvious complexities that arise when they are used in
1975; Swets et al., 2000). ROC analysis has been extended research. There are also common misconceptions and pit-
for use in visualizing and analyzing the behavior of diag- falls when using them in practice. This article attempts to
nostic systems (Swets, 1988). The medical decision making serve as a basic introduction to ROC graphs and as a guide
community has an extensive literature on the use of ROC for using them in research. The goal of this article is to
graphs for diagnostic testing (Zou, 2002). Swets et al. advance general knowledge about ROC graphs so as to
(2000) brought ROC curves to the attention of the wider promote better evaluation practices in the field.
public with their Scientific American article.
One of the earliest adopters of ROC graphs in machine 2. Classifier performance
learning was Spackman (1989), who demonstrated the
value of ROC curves in evaluating and comparing algo- We begin by considering classification problems using
rithms. Recent years have seen an increase in the use of only two classes. Formally, each instance I is mapped to
ROC graphs in the machine learning community, due in one element of the set {p, n} of positive and negative class
part to the realization that simple classification accuracy labels. A classification model (or classifier) is a mapping
is often a poor metric for measuring performance (Provost from instances to predicted classes. Some classification
and Fawcett, 1997; Provost et al., 1998). In addition to models produce a continuous output (e.g., an estimate of
being a generally useful performance graphing method, an instances class membership probability) to which differ-
they have properties that make them especially useful for ent thresholds may be applied to predict class membership.
Other models produce a discrete class label indicating only
E-mail addresses: [email protected], [email protected] the predicted class of the instance. To distinguish between

0167-8655/$ - see front matter  2005 Elsevier B.V. All rights reserved.
doi:10.1016/j.patrec.2005.10.010
862 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874

True class
p n

True False
Y Positives Positives
Hypothesized
class
False True
N Negatives Negatives

Column totals: P N

Fig. 1. Confusion matrix and common performance metrics calculated from it.

the actual class and the predicted class we use the labels 3. ROC space
{Y, N} for the class predictions produced by a model.
Given a classifier and an instance, there are four possible ROC graphs are two-dimensional graphs in which tp
outcomes. If the instance is positive and it is classified as rate is plotted on the Y axis and fp rate is plotted on the
positive, it is counted as a true positive; if it is classified X axis. An ROC graph depicts relative tradeoffs between
as negative, it is counted as a false negative. If the instance benefits (true positives) and costs (false positives). Fig. 2
is negative and it is classified as negative, it is counted as a shows an ROC graph with five classifiers labeled A through
true negative; if it is classified as positive, it is counted as a E.
false positive. Given a classifier and a set of instances (the A discrete classifier is one that outputs only a class label.
test set), a two-by-two confusion matrix (also called a con- Each discrete classifier produces an (fp rate, tp rate) pair
tingency table) can be constructed representing the disposi- corresponding to a single point in ROC space. The classifi-
tions of the set of instances. This matrix forms the basis for ers in Fig. 2 are all discrete classifiers.
many common metrics. Several points in ROC space are important to note. The
Fig. 1 shows a confusion matrix and equations of several lower left point (0, 0) represents the strategy of never issu-
common metrics that can be calculated from it. The num- ing a positive classification; such a classifier commits no
bers along the major diagonal represent the correct deci- false positive errors but also gains no true positives. The
sions made, and the numbers of this diagonal represent opposite strategy, of unconditionally issuing positive classi-
the errors—the confusion—between the various classes. fications, is represented by the upper right point (1, 1).
The true positive rate1 (also called hit rate and recall) of a The point (0, 1) represents perfect classification. Ds per-
classifier is estimated as formance is perfect as shown.
Informally, one point in ROC space is better than
Positives correctly classified another if it is to the northwest (tp rate is higher, fp rate
tp rate 
Total positives is lower, or both) of the first. Classifiers appearing on the
left-hand side of an ROC graph, near the X axis, may be
The false positive rate (also called false alarm rate) of the
classifier is
1.0
Negatives incorrectly classified D
fp rate 
Total negatives B
0.8

Additional terms associated with ROC curves are C

A
True positive rate

0.6
sensitivity ¼ recall
True negatives
specificity ¼ 0.4
False positives þ True negatives
¼ 1  fp rate E
0.2
positive predictive value ¼ precision

0 0.2 0.4 0.6 0.8 1.0

1
False positive rate
For clarity, counts such as TP and FP will be denoted with upper-case
letters and rates such as tp rate will be denoted with lower-case. Fig. 2. A basic ROC graph showing five discrete classifiers.
 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874 863

thought of as ‘‘conservative’’: they make positive classifica- turn corresponds to one ROC point. Thus, a discrete clas-
tions only with strong evidence so they make few false posi- sifier produces only a single point in ROC space.
tive errors, but they often have low true positive rates as Some classifiers, such as a Naive Bayes classifier or a
well. Classifiers on the upper right-hand side of an ROC neural network, naturally yield an instance probability or
graph may be thought of as ‘‘liberal’’: they make positive score, a numeric value that represents the degree to which
classifications with weak evidence so they classify nearly an instance is a member of a class. These values can be
all positives correctly, but they often have high false posi- strict probabilities, in which case they adhere to standard
tive rates. In Fig. 2, A is more conservative than B. Many theorems of probability; or they can be general, uncali-
real world domains are dominated by large numbers of brated scores, in which case the only property that holds
negative instances, so performance in the far left-hand side is that a higher score indicates a higher probability. We
of the ROC graph becomes more interesting. shall call both a probabilistic classifier, in spite of the fact
that the output may not be a proper probability.2
3.1. Random performance Such a ranking or scoring classifier can be used with a
threshold to produce a discrete (binary) classifier: if the
The diagonal line y = x represents the strategy of ran- classifier output is above the threshold, the classifier pro-
domly guessing a class. For example, if a classifier ran- duces a Y, else a N. Each threshold value produces a differ-
domly guesses the positive class half the time, it can be ent point in ROC space. Conceptually, we may imagine
expected to get half the positives and half the negatives varying a threshold from 1 to +1 and tracing a curve
correct; this yields the point (0.5, 0.5) in ROC space. If it through ROC space. Computationally, this is a poor way
guesses the positive class 90% of the time, it can be of generating an ROC curve, and the next section describes
expected to get 90% of the positives correct but its false a more efficient and careful method.
positive rate will increase to 90% as well, yielding Fig. 3 shows an example of an ROC ‘‘curve’’ on a test
(0.9, 0.9) in ROC space. Thus a random classifier will pro- set of 20 instances. The instances, 10 positive and 10 nega-
duce a ROC point that ‘‘slides’’ back and forth on the dia- tive, are shown in the table beside the graph. Any ROC
gonal based on the frequency with which it guesses the curve generated from a finite set of instances is actually a
positive class. In order to get away from this diagonal into step function, which approaches a true curve as the number
the upper triangular region, the classifier must exploit some of instances approaches infinity. The step function in Fig. 3
information in the data. In Fig. 2, Cs performance is virtu- is taken from a very small instance set so that each points
ally random. At (0.7, 0.7), C may be said to be guessing the derivation can be understood. In the table of Fig. 3, the
positive class 70% of the time. instances are sorted by their scores, and each point in the
Any classifier that appears in the lower right triangle ROC graph is labeled by the score threshold that produces
performs worse than random guessing. This triangle is it. A threshold of +1 produces the point (0, 0). As we
therefore usually empty in ROC graphs. If we negate a lower the threshold to 0.9 the first positive instance is clas-
classifier—that is, reverse its classification decisions on sified positive, yielding (0, 0.1). As the threshold is further
every instance—its true positive classifications become false reduced, the curve climbs up and to the right, ending up
negative mistakes, and its false positives become true neg- at (1, 1) with a threshold of 0.1. Note that lowering this
atives. Therefore, any classifier that produces a point in threshold corresponds to moving from the ‘‘conservative’’
the lower right triangle can be negated to produce a point to the ‘‘liberal’’ areas of the graph.
in the upper left triangle. In Fig. 2, E performs much worse Although the test set is very small, we can make some
than random, and is in fact the negation of B. Any classifier tentative observations about the classifier. It appears to
on the diagonal may be said to have no information about perform better in the more conservative region of the
the class. A classifier below the diagonal may be said to graph; the ROC point at (0.1, 0.5) produces its highest
have useful information, but it is applying the information accuracy (70%). This is equivalent to saying that the classi-
incorrectly (Flach and Wu, 2003). fier is better at identifying likely positives than at identify-
Given an ROC graph in which a classifiers performance ing likely negatives. Note also that the classifiers best
appears to be slightly better than random, it is natural to accuracy occurs at a threshold of P0.54, rather than at
ask: ‘‘is this classifiers performance truly significant or is P0.5 as we might expect with a balanced distribution.
it only better than random by chance?’’ There is no conclu- The next section discusses this phenomenon.
sive test for this, but Forman (2002) has shown a method-
ology that addresses this question with ROC curves. 4.1. Relative versus absolute scores

4. Curves in ROC space An important point about ROC graphs is that they mea-
sure the ability of a classifier to produce good relative
Many classifiers, such as decision trees or rule sets, are
designed to produce only a class decision, i.e., a Y or N
on each instance. When such a discrete classifier is applied 2
Techniques exist for converting an uncalibrated score into a proper
to a test set, it yields a single confusion matrix, which in probability but this conversion is unnecessary for ROC curves.
864 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874

is indeed perfect in this ability. The accuracy metric

Inst# Class Score Inst# Class Score
imposes a threshold (score > 0.5) and measures the result-
1 p .9 11 p .4 ing classifications with respect to the scores. The accuracy
measure would be appropriate if the scores were proper
2 p .8 12 n .39
probabilities, but they are not. Another way of saying this
3 n .7 13 p .38 is that the scores are not properly calibrated, as true prob-
4 p .6 14 n .37 abilities are. In ROC space, the imposition of a 0.5 thres-
hold results in the performance designated by the circled
5 p .55 15 n .36 ‘‘accuracy point’’ in Fig. 4. This operating point is subop-
6 p .54 16 n .35 timal. We could use the training set to estimate a prior for
p(p) = 6/10 = 0.6 and use this as a threshold, but it would
7 n .53 17 p .34 still produce suboptimal performance (90% accuracy).
8 n .52 18 n .33 One way to eliminate this phenomenon is to calibrate
the classifier scores. There are some methods for doing this
9 p .51 19 p .30 (Zadrozny and Elkan, 2001). Another approach is to use
10 n .505 20 n .1 an ROC method that chooses operating points based on
their relative performance, and there are methods for doing
.30 .1 this as well (Provost and Fawcett, 1998, 2001). These latter
1
.34 .33 methods are discussed briefly in Section 6.
0.9
A consequence of relative scoring is that classifier scores
.38 .37 .36 .35
0.8 should not be compared across model classes. One model
.4 .39 class may be designed to produce scores in the range
0.7
[0, 1] while another produces scores in [1, +1] or [1, 100].
True positive rate

.51 .505
0.6 Comparing model performance at a common threshold will
.54 .53 .52 be meaningless.
0.5
.55
0.4 4.2. Class skew
.6
0.3
ROC curves have an attractive property: they are insen-
.8 .7
0.2 sitive to changes in class distribution. If the proportion of
.9 positive to negative instances changes in a test set, the
0.1
ROC curves will not change. To see why this is so, consider
Infinity
0 the confusion matrix in Fig. 1. Note that the class distribu-
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False positive rate
tion—the proportion of positive to negative instances—is
the relationship of the left (+) column to the right () col-
Fig. 3. The ROC ‘‘curve’’ created by thresholding a test set. The table umn. Any performance metric that uses values from both
shows 20 data and the score assigned to each by a scoring classifier. The columns will be inherently sensitive to class skews. Metrics
graph shows the corresponding ROC curve with each point labeled by the
threshold that produces it. such as accuracy, precision, lift and F score use values from
both columns of the confusion matrix. As a class distribu-
tion changes these measures will change as well, even if the
fundamental classifier performance does not. ROC graphs
instance scores. A classifier need not produce accurate, cal- are based upon tp rate and fp rate, in which each dimension
ibrated probability estimates; it need only produce relative is a strict columnar ratio, so do not depend on class
accurate scores that serve to discriminate positive and neg- distributions.
ative instances. To some researchers, large class skews and large changes
Consider the simple instance scores shown in Fig. 4, in class distributions may seem contrived and unrealistic.
which came from a Naive Bayes classifier. Comparing the However, class skews of 101 and 102 are very common in
hypothesized class (which is Y if score > 0.5, else N) against real world domains, and skews up to 106 have been
the true classes, we can see that the classifier gets instances observed in some domains (Clearwater and Stern, 1991;
7 and 8 wrong, yielding 80% accuracy. However, consider Fawcett and Provost, 1996; Kubat et al., 1998; Saitta and
the ROC curve on the left side of the figure. The curve rises Neri, 1998). Substantial changes in class distributions are
vertically from (0, 0) to (0, 1), then horizontally to (1, 1). not unrealistic either. For example, in medical decision
This indicates perfect classification performance on this test making epidemics may cause the incidence of a disease to
set. Why is there a discrepancy? increase over time. In fraud detection, proportions of fraud
The explanation lies in what each is measuring. The varied significantly from month to month and place to
ROC curve shows the ability of the classifier to rank the place (Fawcett and Provost, 1997). Changes in a manufac-
positive instances relative to the negative instances, and it turing practice may cause the proportion of defective units
 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874 865

Inst Class Score

no. True Hyp
1
1 p Y 0.99999
2 p Y 0.99999
0.8
True positive rate Accuracy point (threshold = 0.5) 3 p Y 0.99993
0.6 4 p Y 0.99986
Accuracy point (threshold = 0.6)
5 p Y 0.99964
0.4
6 p Y 0.99955
0.2 7 n Y 0.68139
8 n Y 0.50961
0
0 0.2 0.4 0.6 0.8 1 9 n N 0.48880
False positive rate
10 n N 0.44951
Fig. 4. Scores and classifications of 10 instances, and the resulting ROC curve.

produced by a manufacturing line to increase or decrease. assumed; however, they are also used in dynamic environ-
In each of these examples the prevalence of a class may ments such as web page retrieval, where the number of
change drastically without altering the fundamental char- pages irrelevant to a query (N) is many orders of magni-
acteristic of the class, i.e., the target concept. tude greater than P and probably increases steadily over
Precision and recall are common in information retrie- time as web pages are created.
val for evaluating retrieval (classification) performance To see the effect of class skew, consider the curves in
(Lewis, 1990, 1991). Precision-recall graphs are commonly Fig. 5, which show two classifiers evaluated using ROC
used where static document sets can sometimes be curves and precision-recall curves. In Fig. 5a and b, the test

1 1
‘insts.roc.+’
‘insts2.roc.+’
0.8 0.8
‘insts.precall.+’
‘insts2.precall.+’
0.6 0.6

0.4 0.4

0.2 0.2

0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
(a) (b)
1 1
‘instsx10.precall.+’
‘instsx10.roc.+’ ‘insts2x10.precall.+’
0.8 ‘insts2x10.roc.+’ 0.8

0.6 0.6

0.4 0.4

0.2 0.2

0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
(c) (d)

Fig. 5. ROC and precision-recall curves under class skew. (a) ROC curves, 1:1; (b) precision-recall curves, 1:1; (c) ROC curves, 1:10 and (d) precision-
recall curves, 1:10.
866 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874

set has a balanced 1:1 class distribution. Graph 5c and d can simply sort the test instances decreasing by f scores
shows the same two classifiers on the same domain, but and move down the list, processing one instance at a time
the number of negative instances has been increased 10- and updating TP and FP as we go. In this way an ROC
fold. Note that the classifiers and the underlying concept graph can be created from a linear scan.
has not changed; only the class distribution is different. The algorithm is shown in Algorithm 1. TP and FP
Observe that the ROC graphs in Fig. 5a and c are identical, both start at zero. For each positive instance we increment
while the precision-recall graphs in Fig. 5b and d differ sub- TP and for every negative instance we increment FP. We
stantially. In some cases, the conclusion of which classifier maintain a stack R of ROC points, pushing a new point
has superior performance can change with a shifted onto R after each instance is processed. The final output
distribution. is the stack R, which will contain points on the ROC
curve.
4.3. Creating scoring classifiers Let n be the number of points in the test set. This algo-
rithm requires an O(n log n) sort followed by an O(n) scan
Many classifier models are discrete: they are designed down the list, resulting in O(n log n) total complexity.
to produce only a class label from each test instance. Statements 7–10 need some explanation. These are
However, we often want to generate a full ROC curve from necessary in order to correctly handle sequences of equally
a classifier instead of just a single point. To this end we scored instances. Consider the ROC curve shown in Fig. 6.
want to generate scores from a classifier rather than just Assume we have a test set in which there is a sequence of
a class label. There are several ways of producing such instances, four negatives and six positives, all scored
scores. equally by f. The sort in line 1 of Algorithm 1 does not
Many discrete classifier models may easily be converted impose any specific ordering on these instances since their
to scoring classifiers by ‘‘looking inside’’ them at the f scores are equal. What happens when we create an
instance statistics they keep. For example, a decision tree ROC curve? In one extreme case, all the positives end up
determines a class label of a leaf node from the proportion at the beginning of the sequence and we generate the ‘‘opti-
of instances at the node; the class decision is simply the mistic’’ upper L segment shown in Fig. 6. In the opposite
most prevalent class. These class proportions may serve
as a score (Provost and Domingos, 2001). A rule learner
keeps similar statistics on rule confidence, and the confi-
dence of a rule matching an instance can be used as a score Algorithm 1. Efficient method for generating ROC points
(Fawcett, 2001). Inputs: L, the set of test examples; f(i), the probabilistic
Even if a classifier only produces a class label, an classifiers estimate that example i is positive; P and N, the
aggregation of them may be used to generate a score. number of positive and negative examples.
MetaCost (Domingos, 1999) employs bagging to generate Outputs: R, a list of ROC points increasing by fp rate.
an ensemble of discrete classifiers, each of which produces Require: P > 0 and N > 0
a vote. The set of votes could be used to generate a 1: Lsorted L sorted decreasing by f scores
score.3 2: FP TP 0
Finally, some combination of scoring and voting can be 3: R hi
employed. For example, rules can provide basic probability 4: fprev 1
estimates, which may then be used in weighted voting 5: i 1
(Fawcett, 2001). 6: while i 6 jLsortedj do
7: if f(i) 5 fprev then
5. Efficient generation of ROC curves  
FP TP
8: push ; onto R
N P
Given a test set, we often want to generate an ROC
9: fprev f(i)
curve efficiently from it. We can exploit the monotonicity
of thresholded classifications: any instance that is classified 10: end if
positive with respect to a given threshold will be classified 11: if Lsorted[i] is a positive example then
positive for all lower thresholds as well. Therefore, we 12: TP TP + 1
13: else /* i is a negative example */
14: FP FP + 1
15: end if
3
MetaCost actually works in the opposite direction because its goal is to 16: i i+1
generate a discrete classifier. It first creates a probabilistic classifier, then 17: end while
applies knowledge of the error costs and class skews to relabel the  
instances so as to ‘‘optimize’’ their classifications. Finally, it learns a FP TP
18: push ; onto R /* This is (1,1) */
specific discrete classifier from this new instance set. Thus, MetaCost is not N P
a good method for creating a scoring classifier, though its bagging method
may be.
19: end
 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874 867

1.0 invariant with respect to the operating conditions (class skew

Optimistic
and error costs). As these conditions change, the region of
0.8
interest may change, but the graph itself will not.
Provost and Fawcett (1998, 2001) show that a set of
operating conditions may be transformed easily into a
True positive rate

Expected
0.6 so-called iso-performance line in ROC space. Two points
in ROC space, (FP1, TP1) and (FP2, TP2), have the same
0.4 performance if
Pessimistic

TP 2  TP 1 cðY; nÞpðnÞ
0.2 ¼ ¼m ð1Þ
FP 2  FP 1 cðN; pÞpðpÞ

0 This equation defines the slope of an iso-performance line.

0 0.2 0.4 0.6 0.8 1.0 All classifiers corresponding to points on a line of slope m
False positive rate have the same expected cost. Each set of class and cost dis-
Fig. 6. The optimistic, pessimistic and expected ROC segments resulting
tributions defines a family of iso-performance lines. Lines
from a sequence of 10 equally scored instances. ‘‘more northwest’’ (having a larger TP-intercept) are better
because they correspond to classifiers with lower expected
cost. More generally, a classifier is potentially optimal if
and only if it lies on the convex hull of the set of points
extreme, all the negatives end up at the beginning of the in ROC space. The convex hull of the set of points in
sequence and we get the ‘‘pessimistic’’ lower L shown in ROC space is called the ROC convex hull (ROCCH) of
Fig. 6. Any mixed ordering of the instances will give a dif- the corresponding set of classifiers.
ferent set of step segments within the rectangle formed by Fig. 7a shows four ROC curves (A through D) and their
these two extremes. However, the ROC curve should repre- convex hull (labeled CH). D is not on the convex hull and is
sent the expected performance of the classifier, which, lack- clearly sub-optimal. B is also not optimal for any condi-
ing any other information, is the average of the pessimistic tions because it is not on the convex hull either. The convex
and optimistic segments. This average is the diagonal of the hull is bounded only by points from curves A and C. Thus,
rectangle, and can be created in the ROC curve algorithm if we are seeking optimal classification performance, classi-
by not emitting an ROC point until all instances of equal f fiers B and D may be removed entirely from consideration.
values have been processed. This is what the fprev variable In addition, we may remove any discrete points from A and
and the if statement of line 7 accomplish. C that are not on the convex hull.
Instances that are scored equally may seem unusual Fig. 7b shows the A and C curves again with two explicit
but with some classifier models they are common. For iso-performance lines, a and b. Consider a scenario in
example, if we use instance counts at nodes in a decision which negatives outnumber positives by 10 to 1, but false
tree to score instances, a large, high-entropy leaf node positives and false negatives have equal cost. By Eq. (1)
may produce many equally scored instances of both clas- m = 10, and the most northwest line of slope m = 10 is a,
ses. If such instances are not averaged, the resulting ROC tangent to classifier A, which would be the best performing
curves will be sensitive to the test set ordering, and different classifier for these conditions.
orderings can yield very misleading curves. This can be Consider another scenario in which the positive and
especially critical in calculating the area under an ROC negative example populations are evenly balanced but a
curve, discussed in Section 7. Consider a decision tree con- false negative is 10 times as expensive as a false positive.
taining a leaf node accounting for n positives and m nega- By Eq. (1) m = 1/10. The most northwest line of slope 1/
tives. Every instance that is classified to this leaf node will 10 would be line b, tangent to classifier C. C is the optimal
be assigned the same score. The rectangle of Fig. 6 will be classifier for these conditions.
nm
of size PN , and if these instances are not averaged this one If we wanted to generate a classifier somewhere on the
leaf may account for errors in ROC curve area as high convex hull between A and C, we could interpolate
nm
as 2PN . between the two. Section 10 explains how to generate such
a classifier.
6. The ROC convex hull This ROCCH formulation has a number of useful
implications. Since only the classifiers on the convex hull
One advantage of ROC graphs is that they enable visual- are potentially optimal, no others need be retained. The
izing and organizing classifier performance without regard operating conditions of the classifier may be translated into
to class distributions or error costs. This ability becomes very an iso-performance line, which in turn may be used to iden-
important when investigating learning with skewed distribu- tify a portion of the ROCCH. As conditions change, the
tions or cost-sensitive learning. A researcher can graph the hull itself does not change; only the portion of interest
performance of a set of classifiers, and that graph will remain will.
868 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874

1.0
1.0
C
β

0.8 C
CH 0.8

α
True positive rate

True positive rate

0.6 0.6
A
A
D
0.4 0.4

0.2 0.2

0
0
0 0.2 0.4 0.6 0.8 1.0 0 0.2 0.4 0.6 0.8 1.0
(a) False positive rate (b) False positive rate

Fig. 7. (a) The ROC convex hull identifies potentially optimal classifiers. (b) Lines a and b show the optimal classifier under different sets of conditions.

7. Area under an ROC curve (AUC) equivalent to the Wilcoxon test of ranks (Hanley and
McNeil, 1982). The AUC is also closely related to the Gini
An ROC curve is a two-dimensional depiction of classi- coefficient (Breiman et al., 1984), which is twice the area
fier performance. To compare classifiers we may want to between the diagonal and the ROC curve. Hand and Till
reduce ROC performance to a single scalar value represent- (2001) point out that Gini + 1 = 2 · AUC.
ing expected performance. A common method is to calcu- Fig. 8a shows the areas under two ROC curves, A and
late the area under the ROC curve, abbreviated AUC B. Classifier B has greater area and therefore better average
(Bradley, 1997; Hanley and McNeil, 1982). Since the performance. Fig. 8b shows the area under the curve of a
AUC is a portion of the area of the unit square, its value binary classifier A and a scoring classifier B. Classifier A
will always be between 0 and 1.0. However, because ran- represents the performance of B when B is used with a sin-
dom guessing produces the diagonal line between (0, 0) gle, fixed threshold. Though the performance of the two is
and (1, 1), which has an area of 0.5, no realistic classifier equal at the fixed point (As threshold), As performance
should have an AUC less than 0.5. becomes inferior to B further from this point.
The AUC has an important statistical property: the It is possible for a high-AUC classifier to perform worse
AUC of a classifier is equivalent to the probability that in a specific region of ROC space than a low-AUC classi-
the classifier will rank a randomly chosen positive instance fier. Fig. 8a shows an example of this: classifier B is gener-
higher than a randomly chosen negative instance. This is ally better than A except at FPrate > 0.6 where A has a

1.0 1.0
B
B

0.8 0.8 A

A
True positive rate

True positive rate

0.6 0.6

0.4 0.4

0.2 0.2

0 0
0 0.2 0.4 0.6 0.8 1.0 0 0.2 0.4 0.6 0.8 1.0
(a) False positive rate (b) False positive rate

Fig. 8. Two ROC graphs. The graph on the left shows the area under two ROC curves. The graph on the right shows the area under the curves of a
discrete classifier (A) and a probabilistic classifier (B).
 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874 869

classifiers simply by graphing them in ROC space and see-

Algorithm 2. Calculating the area under an ROC curve
ing which ones dominate. This is misleading; it is analo-
Inputs: L, the set of test examples; f(i), the probabilistic
gous to taking the maximum of a set of accuracy figures
classifiers estimate that example i is positive; P and N, the
from a single test set. Without a measure of variance we
number of positive and negative examples.
cannot compare the classifiers.
Outputs: A, the area under the ROC curve.
Averaging ROC curves is easy if the original instances
Require: P > 0 and N > 0
are available. Given test sets T1, T2, . . . , Tn, generated from
1: Lsorted L sorted decreasing by f scores
cross-validation or the bootstrap method, we can simply
2: FP TP 0
merge sort the instances together by their assigned scores
3: FPprev TPprev 0
into one large test set TM. We then run an ROC curve gen-
4: A 0
eration algorithm such as algorithm 1 on TM and plot the
5: fprev 1
result. However, the primary reason for using multiple test
6: i 1
sets is to derive a measure of variance, which this simple
7: while i 6 jLsortedj do
merging does not provide. We need a more sophisticated
8: if f(i) 5 fprev then
method that samples individual curves at different points
9: A A + TRAPEZOID_AREA(FP, FPprev,
and averages the samples.
TP, TPprev)
ROC space is two-dimensional, and any average is nec-
10: fprev f(i)
essarily one-dimensional. ROC curves can be projected
11: FPprev FP
onto a single dimension and averaged conventionally, but
12: TPprev TP
this leads to the question of whether the projection is
13: end if
appropriate, or more precisely, whether it preserves charac-
14: if i is a positive example then
teristics of interest. The answer depends upon the reason
15: TP TP + 1
for averaging the curves. This section presents two methods
16: else /* i is a negative example */
for averaging ROC curves: vertical and threshold aver-
17: FP FP + 1
aging.
18: end if
Fig. 9a shows five ROC curves to be averaged. Each
19: i i+1
contains a thousand points and has some concavities.
20: end while
Fig. 9b shows the curve formed by merging the five test sets
21: A A + TRAPEZOID_AREA(N, FPprev, N, TPprev)
and computing their combined ROC curve. Fig. 9c and d
22: A A/(P · N) /* scale from P · N onto the unit
shows average curves formed by sampling the five individ-
square */
ual ROC curves. The error bars are 95% confidence
23: end
intervals.
1: function TRAPEZOID_AREA(X1, X2, Y1, Y2)
2: Base jX1  X2j
8.1. Vertical averaging
3: Heightavg (Y1 + Y2)/2
4: return Base · Heightavg
Vertical averaging takes vertical samples of the ROC
5: end function
curves for fixed FP rates and averages the corresponding
TP rates. Such averaging is appropriate when the FP
rate can indeed be fixed by the researcher, or when a
slight advantage. But in practice the AUC performs very single-dimensional measure of variation is desired. Pro-
well and is often used when a general measure of predic- vost et al. (1998) used this method in their work of
tiveness is desired. averaging ROC curves of a classifier for k-fold cross-
The AUC may be computed easily using a small modi- validation.
fication of algorithm 1, shown in Algorithm 2. Instead of In this method each ROC curve is treated as a function,
collecting ROC points, the algorithm adds successive areas Ri, such that tp rate = Ri(fp rate). This is done by choosing
of trapezoids to A. Trapezoids are used rather than rectan- the maximum tp rate for each fp rate and interpolating
gles in order to average the effect between points, as between points when necessary. The averaged ROC curve
illustrated in Fig. 6. Finally, the algorithm divides A by ^
is the function Rðfp rateÞ ¼ mean½Ri ðfp rateÞ. To plot an
the total possible area to scale the value to the unit average ROC curve we can sample from R ^ at points regu-
square. larly spaced along the fp rate-axis. Confidence intervals of
the mean of tp rate are computed using the common
8. Averaging ROC curves assumption of a binomial distribution.
Algorithm 3 computes this vertical average of a
Although ROC curves may be used to evaluate classifi- set of ROC points. It leaves the means in the array
ers, care should be taken when using them to make conclu- TPavg.
sions about classifier superiority. Some researchers have Several extensions have been left out of this algorithm
assumed that an ROC graph may be used to select the best for clarity. The algorithm may easily be extended to
870 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874

1 1

0.8 0.8

True positive rate

True positive rate

0.6 0.6

0.4 0.4

0.2 0.2

0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
False positive rate False positive rate
(a) (b)
1 1

0.8 0.8
True positive rate

True positive rate

0.6 0.6

0.4 0.4

0.2 0.2

0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
False positive rate False positive rate
(c) (d)

Fig. 9. ROC curve averaging. (a) ROC curves of five instance samples, (b) ROC curve formed by merging the five samples, (c) the curves of a averaged
vertically and (d) the curves of a averaged by threshold.

compute standard deviations of the samples in order to tical averaging does, it samples based on the thresholds
draw confidence bars. Also, the function TP_FOR_FP may that produced these points. The method must generate a
be optimized somewhat. Because it is only called on mono- set of thresholds to sample, then for each threshold it finds
tonically increasing values of FP, it need not scan each the corresponding point of each ROC curve and averages
ROC array from the beginning every time; it could keep them.
a record of the last point seen and initialize i from this Algorithm 4 shows the basic method for doing this. It
array. generates an array T of classifier scores which are sorted
Fig. 9c shows the vertical average of the five curves in from largest to smallest and used as the set of thresholds.
Fig. 9a. The vertical bars on the curve show the 95% con- These thresholds are sampled at fixed intervals determined
fidence region of the ROC mean. For this average curve, by samples, the number of samples desired. For a given
the curves were sampled at FP rates from 0 through 1 by threshold, the algorithm selects from each ROC curve the
0.1. It is possible to sample curves much more finely but point of greatest score less than or equal to the threshold.4
the confidence bars may become difficult to read. These points are then averaged separately along their X
and Y axes, with the center point returned in the Avg array.
8.2. Threshold averaging Fig. 9d shows the result of averaging the five curves of
Fig. 9a by thresholds. The resulting curve has average
Vertical averaging has the advantage that averages are points and confidence bars in the X and Y directions.
made of a single dependent variable, the true positive rate, The bars shown are at the 95% confidence level.
which simplifies computing confidence intervals. However, There are some minor limitations of threshold averaging
Holte (2002) has pointed out that the independent variable, with respect to vertical averaging. To perform threshold
false positive rate, is often not under the direct control of averaging we need the classifier score assigned to each
the researcher. It may be preferable to average ROC points point. Also, Section 4.1 pointed out that classifier scores
using an independent variable whose value can be con-
trolled directly, such as the threshold on the classifier scores.
Threshold averaging accomplishes this. Instead of sam- 4
We assume the ROC points have been generated by an algorithm like 1
pling points based on their positions in ROC space, as ver- that deals correctly with equally scored instances.
 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874 871

Algorithm 3. Vertical averaging of ROC curves Algorithm 4. Threshold averaging of ROC curves
Inputs: samples, the number of FP samples; nrocs, the Inputs: samples, the number of threshold samples; nrocs,
number of ROC curves to be sampled, ROCS[nrocs], an the number of ROC curves to be sampled; ROCS[nrocs], an
array of nrocs ROC curves; npts[m], the number of points array of nrocs ROC curves sorted by score; npts[m], the
in ROC curve m. Each ROC point is a structure of two number of points in ROC curve m. Each ROC point is a
members, the rates fpr and tpr. structure of three members, fpr, tpr and score.
Output: Array tpravg[samples + 1], containing the vertical Output: Avg[samples + 1], an array of (X, Y) points
averages. constituting the average ROC curve.
1: s 1 Require: samples > 1
2: for fprsample = 0 to 1 by 1/samples do 1: initialize array T to contain all scores of all ROC
3: tprsum 0 points
4: for i = 1 to nrocs do 2: sort T in descending order
5: tprsum tprsum + TPR_FOR_FPR(fprsample, 3: s 1
ROCS[i], npts[i]) 4: for tidx = 1 to length(T) by int(length(T)/samples) do
6: end for 5: fprsum 0
7: tpravg[s] tprsum/nrocs 6: tprsum 0
8: s s+1 7: for i = 1 to nrocs do
9: end for 8: p ROC_POINT_AT_THRESHOLD(ROCS[i], npts[i],
10: end T[tidx])
1: function TPR_FOR_FPR(fprsample, ROC, npts) 9: fprsum fprsum + p.fpr
2: i 1 10: tprsum tprsum + p.tpr
3: while i < npts and ROC [i + 1].fpr 6 fprsample do 11: end for
4: i i+1 12: Avg[s] (fprsum/nrocs, tprsum/nrocs)
5: end while 13: s s+1
6: if ROC[i].fpr = fprsample then 14: end for
7: return ROC[i].tpr 15: end
8: else 1: function ROC_POINT_AT_THRESHOLD(ROC, npts, thresh)
9: return INTERPOLATE(ROC[i], ROC [i + 1], fprsample) 2: i 1
10: end if 3: while i 6 npts and ROC[i]. score > thresh do
11: end function 4: i i+1
1: function INTERPOLATE(ROCP1, ROCP2, X) 5: end while
2: slope = (ROCP2.tpr  ROCP1.tpr)/(ROCP2.fpr  6: return ROC[i]
ROCP1.fpr) 7: end function
3: return ROCP1.tpr + slope Æ (X  ROCP1.fpr)
4: end function
ical decision making in which two-class diagnostic prob-
lems—presence or absence of an abnormal condition—
should not be compared across model classes. Because of are common. The two axes represent tradeoffs between
this, ROC curves averaged from different model classes errors (false positives) and benefits (true positives) that a
may be misleading because the scores may be incom- classifier makes between two classes. Much of the analysis
mensurate. is straightforward because of the symmetry that exists in
Finally, Macskassy and Provost (2004) have investi- the two-class problem. The resulting performance can be
gated different techniques for generating confidence bands graphed in two dimensions, which is easy to visualize.
for ROC curves. They investigate confidence intervals from
vertical and threshold averaging, as well as three methods 9.1. Multi-class ROC graphs
from the medical field for generating bands (simultaneous
join confidence regions, Working-Hotelling based bands, With more than two classes the situation becomes much
and fixed-width confidence bands). The reader is referred more complex if the entire space is to be managed. With n
to their paper for a much more detailed discussion of the classes the confusion matrix becomes an n · n matrix con-
techniques, their assumptions, and empirical studies. taining the n correct classifications (the major diagonal
entries) and n2  n possible errors (the off-diagonal entries).
9. Decision problems with more than two classes Instead of managing trade-offs between TP and FP, we
have n benefits and n2  n errors. With only three classes,
Discussions up to this point have dealt with only two the surface becomes a 32  3 = 6-dimensional polytope.
classes, and much of the ROC literature maintains this Lane (2000) has outlined the issues involved and the pros-
assumption. ROC analysis is commonly employed in med- pects for addressing them. Srinivasan (1999) has shown
872 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874

that the analysis behind the ROC convex hull extends to desired a measure that is insensitive to class distribution
multiple classes and multi-dimensional convex hulls. and error costs. The derivation is too detailed to summa-
One method for handling n classes is to produce n differ- rize here, but it is based upon the fact that the AUC is
ent ROC graphs, one for each class. Call this the class ref- equivalent to the probability that the classifier will rank a
erence formulation. Specifically, if C is the set of all classes, randomly chosen positive instance higher than a randomly
ROC graph i plots the classification performance using chosen negative instance. From this probabilistic form,
class ci as the positive class and all other classes as the neg- they derive a formulation that measures the unweighted
ative class, i.e. pairwise discriminability of classes. Their measure, which
P i ¼ ci ð2Þ they call M, is equivalent to:
[ X
Ni ¼ cj 2 C ð3Þ 2
AUCtotal ¼ AUCðci ; cj Þ
j6¼i jCjðjCj  1Þ fc ;c g2C
i j

While this is a convenient formulation, it compromises one

of the attractions of ROC graphs, namely that they are where n is the number of classes and AUC(ci, cj) is the area
insensitive to class skew (see Section 4.2). Because each under the two-class ROC curve involving classes ci and cj.
Ni comprises the union of n  1 classes, changes in preva- The summation is calculated over all pairs of distinct
lence within these classes may alter the cis ROC graph. classes, irrespective of order. There are jCj(jCj  1)/2
For example, assume that some class ck 2 N is particularly such pairs, so the time complexity of their measure is
easy to identify. A classifier for class ci, i 5 k may exploit O(jCj2n log n). While Hand and Tills formulation is well
some characteristic of ck in order to produce low scores for justified and is insensitive to changes in class distribution,
ck instances. Increasing the prevalence of ck might alter the there is no easy way to visualize the surface whose area is
performance of the classifier, and would be tantamount to being calculated.
changing the target concept by increasing the prevalence of
one of its disjuncts. This in turn would alter the ROC 10. Interpolating classifiers
curve. However, with this caveat, this method can work
well in practice and provide reasonable flexibility in Sometimes the performance desired of a classifier is not
evaluation. exactly produced by any available classifier, but lies
between two available classifiers. The desired performance
9.2. Multi-class AUC can be obtained by sampling the decisions of each classifier.
The sampling ratio will determine where the resulting
The AUC is a measure of the discriminability of a pair classification performance lies.
of classes. In a two-class problem, the AUC is a single sca- For a concrete example, consider the decision problem
lar value, but a multi-class problem introduces the issue of of the CoIL Challenge 2000 (van der Putten and Someren,
combining multiple pairwise discriminability values. The 2000). In this challenge there is a set of 4000 clients to
reader is referred to Hand and Tills (2001) article for an whom we wish to market a new insurance policy. Our bud-
excellent discussion of these issues. get dictates that we can afford to market to only 800 of
One approach to calculating multi-class AUCs was them, so we want to select the 800 who are most likely to
taken by Provost and Domingos (2001) in their work on respond to the offer. The expected class prior of responders
probability estimation trees. They calculated AUCs for is 6%, so within the population of 4000 we expect to have
multi-class problems by generating each class reference 240 responders (positives) and 3760 non-responders
ROC curve in turn, measuring the area under the curve, (negatives).
then summing the AUCs weighted by the reference classs Assume we have generated two classifiers, A and B,
prevalence in the data. More precisely, they define which score clients by the probability they will buy the
X policy. In ROC space A lies at (0.1, 0.2) and B lies at
AUCtotal ¼ AUCðci Þ  pðci Þ (0.25, 0.6), as shown in Fig. 10. We want to market to
ci 2C
exactly 800 people so our solution constraint is fp
where AUC(ci) is the area under the class reference ROC rate · 3760 + tp rate · 240 = 800. If we use A we expect
curve for ci, as in Eq. (3). This definition requires only jCj 0.1 · 3760 + 0.2 · 240 = 424 candidates, which is too few.
AUC calculations, so its overall complexity is O(jCjn log n). If we use B we expect 0.25 · 3760 + 0.6 · 240 = 1084
The advantage of Provost and Domingoss AUC formu- candidates, which is too many. We want a classifier
lation is that AUCtotal is generated directly from class ref- between A and B.
erence ROC curves, and these curves can be generated and The solution constraint is shown as a dashed line in
visualized easily. The disadvantage is that the class refer- Fig. 10. It intersects the line between A and B at C, approx-
ence ROC is sensitive to class distributions and error costs, imately (0.18, 0.42). A classifier at point C would give the
so this formulation of AUCtotal is as well. performance we desire and we can achieve it using linear
Hand and Till (2001) take a different approach in their interpolation. Calculate k as the proportional distance that
derivation of a multi-class generalization of the AUC. They C lies on the line between A and B:
 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874 873

1.0 Fawcett, T., 2001. Using rule sets to maximize ROC performance. In:
Proc. IEEE Internat. Conf. on Data Mining (ICDM-2001), pp. 131–
constraint line: 138.
0.8 TPr * 240 + FPr * 3760 = 800 Fawcett, T., Provost, F., 1996. Combining data mining and machine
learning for effective user profiling. In: Simoudis, E., Han, J., Fayyad,
B
U. (Eds.), Proc. Second Internat. Conf. on Knowledge Discovery and
True positive rate

0.6 Data Mining. AAAI Press, Menlo Park, CA, pp. 8–13.
Fawcett, T., Provost, F., 1997. Adaptive fraud detection. Data Mining
C
and Knowledge Discovery 1 (3), 291–316.
0.4 Flach, P., Wu, S., 2003. Repairing concavities in ROC curves. In: Proc.
2003 UK Workshop on Computational Intelligence. University of
Bristol, pp. 38–44.
A
Forman, G., 2002. A method for discovering the insignificance of ones
0.2
} k
best classifier and the unlearnability of a classification task. In: Lavrac,
N., Motoda, H., Fawcett, T. (Eds.), Proc. First Internat. Workshop on
0 Data Mining Lessons Learned (DMLL-2002). Available from: http://
www.purl.org/NET/tfawcett/DMLL-2002/Forman.pdf.
0 0.05 0.1 0.15 0.2 0.25 0.3 Hand, D.J., Till, R.J., 2001. A simple generalization of the area under the
False positive rate ROC curve to multiple class classification problems. Mach. Learning
45 (2), 171–186.
Fig. 10. Interpolating classifiers.
Hanley, J.A., McNeil, B.J., 1982. The meaning and use of the area under a
receiver operating characteristic (ROC) curve. Radiology 143, 29–
0:18  0:1 36.
k¼  0:53 Holte, R., 2002. Personal communication.
0:25  0:1
Kubat, M., Holte, R.C., Matwin, S., 1998. Machine learning for the
Therefore, if we sample Bs decisions at a rate of 0.53 and detection of oil spills in satellite radar images. Machine Learning 30
As decisions at a rate of 1  0.53 = 0.47 we should attain (2–3), 195–215.
Cs performance. In practice this fractional sampling can Lane, T., 2000. Extensions of ROC analysis to multi-class domains. In:
Dietterich, T., Margineantu, D., Provost, F., Turney, P. (Eds.), ICML-
be done by randomly sampling decisions from each: for 2000 Workshop on Cost-Sensitive Learning.
each instance, generate a random number between zero Lewis, D., 1990. Representation quality in text classification: An intro-
and one. If the random number is greater than k, apply duction and experiment. In: Proc. Workshop on Speech and Natu-
classifier A to the instance and report its decision, else pass ral Language. Morgan Kaufmann, Hidden Valley, PA, pp. 288–
the instance to B. 295.
Lewis, D., 1991. Evaluating text categorization. In: Proc. Speech and
Natural Language Workshop. Morgan Kaufmann, pp. 312–318.
11. Conclusion Macskassy, S., Provost, F., 2004. Confidence bands for ROC curves:
Methods and an empirical study. In: Proc. First Workshop on ROC
ROC graphs are a very useful tool for visualizing and Analysis in AI (ROCAI-04).
evaluating classifiers. They are able to provide a richer Provost, F., Domingos, P., 2001. Well-trained PETs: Improving prob-
ability estimation trees, CeDER Working Paper #IS-00-04, Stern
measure of classification performance than scalar measures School of Business, New York University, NY, NY 10012.
such as accuracy, error rate or error cost. Because they de- Provost, F., Fawcett, T., 1997. Analysis and visualization of classifier
couple classifier performance from class skew and error performance: Comparison under imprecise class and cost distributions.
costs, they have advantages over other evaluation measures In: Proc. Third Internat. Conf. on Knowledge Discovery and Data
such as precision-recall graphs and lift curves. However, as Mining (KDD-97). AAAI Press, Menlo Park, CA, pp. 43–48.
Provost, F., Fawcett, T., 1998. Robust classification systems for imprecise
with any evaluation metric, using them wisely requires environments. In: Proc. AAAI-98. AAAI Press, Menlo Park, CA,
knowing their characteristics and limitations. It is hoped pp. 706–713. Available from: <http://www.purl.org/NET/tfawcett/
that this article advances the general knowledge about papers/aaai98-dist.ps.gz>.
ROC graphs and helps to promote better evaluation prac- Provost, F., Fawcett, T., 2001. Robust classification for imprecise
tices in the pattern recognition community. environments. Mach. Learning 42 (3), 203–231.
Provost, F., Fawcett, T., Kohavi, R., 1998. The case against accuracy
estimation for comparing induction algorithms. In: Shavlik, J. (Ed.),
References Proc. ICML-98. Morgan Kaufmann, San Francisco, CA, pp. 445–453.
Available from: <http://www.purl.org/NET/tfawcett/papers/ICML98-
Bradley, A.P., 1997. The use of the area under the ROC curve in the final.ps.gz>.
evaluation of machine learning algorithms. Pattern Recogn. 30 (7), Saitta, L., Neri, F., 1998. Learning in the ‘‘real world’’. Mach. Learning
1145–1159. 30, 133–163.
Breiman, L., Friedman, J., Olshen, R., Stone, C., 1984. Classification and Spackman, K.A., 1989. Signal detection theory: Valuable tools for
Regression Trees. Wadsworth International Group, Belmont, CA. evaluating inductive learning. In: Proc. Sixth Internat. Workshop on
Clearwater, S., Stern, E., 1991. A rule-learning program in high energy Machine Learning. Morgan Kaufman, San Mateo, CA, pp. 160–163.
physics event classification. Comput. Phys. Commun. 67, 159–182. Srinivasan, A., 1999. Note on the location of optimal classifiers in n-
Domingos, P., 1999. MetaCost: A general method for making classifiers dimensional ROC space. Technical Report PRG-TR-2-99, Oxford
cost-sensitive. In: Proc. Fifth ACM SIGKDD Internat. Conf. on University Computing Laboratory, Oxford, England. Available from:
Knowledge Discovery and Data Mining, pp. 155–164. <http://citeseer.nj.nec.com/srinivasan99note.html>.
Egan, J.P., 1975. Signal detection theory and ROC analysis, Series in Swets, J., 1988. Measuring the accuracy of diagnostic systems. Science 240,
Cognition and Perception. Academic Press, New York. 1285–1293.
874 T. Fawcett / Pattern Recognition Letters 27 (2006) 861–874

Swets, J.A., Dawes, R.M., Monahan, J., 2000. Better decisions through Zadrozny, B., Elkan, C., 2001. Obtaining calibrated probability estimates
science. Scientific American 283, 82–87. from decision trees and naive Bayesian classiers. In: Proc. Eighteenth
van der Putten, P., van Someren, M., 2000. CoIL challenge 2000: The Internat. Conf. on Machine Learning, pp. 609–616.
insurance company case. Technical Report 2000–09, Leiden Institute Zou, K.H., 2002. Receiver operating characteristic (ROC) literature
of Advanced Computer Science, Universiteit van Leiden. Available research. On-line bibliography available from: <http://splweb.bwh.
from: <http://www.liacs.nl/putten/library/cc2000>. harvard.edu:8000/pages/ppl/zou/roc.html>.

View publication stats