Scribd
Upload
40 views8 pages

Roc 1 PDF

Uploaded by

Muamer Besic
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
40 views8 pages

Roc 1 PDF

Uploaded by

Muamer Besic
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

CS340 Machine learning

ROC curves

Performance measures for binary classifiers


Confusion matrix, contingency table

precision = positive
predictive value (PPV) = TP / P-hat

Sensitivity = recall =
True pos rate = hit rate
= TP / P = 1-FNR

False pos rate = false acceptance =


= type I error rate = FP / N = 1-spec

False neg rate = false rejection =


type II error rate = FN / P = 1-TPR

Specificity = TN / N = 1-FPR

Performance depends on threshold


Declare xn to be a positive if p(y=1|xn)>, otherwise
declare it to be negative (y=0)
yn = 1 p(y = 1|xn ) >

Number of TPs and FPs depends on threshold .


As we change , we get different (TPR, FPR)
points.

T P R = p(
y = 1|y = 1)
F P R = p(
y = 1|y = 0)

Example
i
1
2
3
4
5
6
7
8
9

i
1
2
3
4
5
6
7
8
9

yi
1
1
1
1
1
0
0
0
0

yi
1
1
1
1
1
0
0
0
0

p(yi = 1|xi )
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1

p(yi = 1|xi )
0.9
0.8
0.7
0.6
0.2
0.6
0.3
0.2
0.1

yi ( = 0)
1
1
1
1
1
1
1
1
1

yi ( = 0)
1
1
1
1
1
1
1
1
1

yi ( = 0.5)
1
1
1
1
1
0
0
0
0

yi ( = 0.5)
1
1
1
1
0
1
0
0
0

yi ( = 1)
0
0
0
0
0
0
0
0
0

yi ( = 1)
0
0
0
0
0
0
0
0
0

Performance measures
EER- Equal error rate/ cross over error rate (false
pos rate = false neg rate), smaller is better
AUC - Area under curve, larger is better
Accuracy = (TP+TN)/(P+N)

Precision-recall curves
Useful when notion of negative (and hence FPR)
is not well defined, or too many negatives (rare
event detection)
Recall = of those that exist, how many did you find?
Precision = of those that you found, how many
correct?
2P R
2
=
F
=
F-score is harmonic mean
1/P + 1/R
R+P
prec = p(y = 1|
y = 1)
recall = p(
y = 1|y = 1)

Word of caution
Consider binary classifiers A, B, C
1
0

A
1
0.9
0

.
0
0.1
0

B
1
0.8
0.1

.
0
0
0.1

C
1
0.78
0.12

.
0
0
0.1

Clearly A is useless, since it always predicts label


1, regardless of the input. Also, B is slightly better
than C (less probability mass wasted on the offdiagonal entries). Yet here are the performance
metrics.
Metric
Accuracy
Precision
Recall
F-score

A
0.9
0.9
1.0
0.947

B
0.9
1.0
0.888
0.941

C
0.88
1.0
0.8667
0.9286

Mutual information is a better measure


The MI between estimated and true label is
I(Y , Y ) =

1
1 

y=0

p(
y, y)
p(
y , y) log
p(
y)p(y)
y=0

This gives the intuitively correct rankings B>C>A


Metric
Accuracy
Precision
Recall
F-score
Mutual information

A
0.9
0.9
1.0
0.947
0

B
0.9
1.0
0.888
0.941
0.1865

C
0.88
1.0
0.8667
0.9286
0.1735

You might also like