Scribd
Upload
49 views8 pages

CS340 Machine Learning ROC Curves

This document discusses performance measures for binary classifiers such as precision, recall, and ROC curves. It explains that precision measures the proportion of predicted positives that are actual positives, while recall measures the proportion of actual positives that are correctly predicted as such. The document also discusses how precision and recall can be visualized using precision-recall curves and ROC curves. Finally, it notes that accuracy, precision, and recall are not always the best metrics, and mutual information may be a better measure in some cases.

Uploaded by

ProgAchr
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
49 views8 pages

CS340 Machine Learning ROC Curves

This document discusses performance measures for binary classifiers such as precision, recall, and ROC curves. It explains that precision measures the proportion of predicted positives that are actual positives, while recall measures the proportion of actual positives that are correctly predicted as such. The document also discusses how precision and recall can be visualized using precision-recall curves and ROC curves. Finally, it notes that accuracy, precision, and recall are not always the best metrics, and mutual information may be a better measure in some cases.

Uploaded by

ProgAchr
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

CS340 Machine learning ROC curves

Performance measures for binary classifiers


Confusion matrix, contingency table

precision = positive predictive value (PPV) = TP / P-hat

Sensitivity = recall = True pos rate = hit rate = TP / P = 1-FNR False neg rate = false rejection = type II error rate = FN / P = 1-TPR

False pos rate = false acceptance = = type I error rate = FP / N = 1-spec

Specificity = TN / N = 1-FPR

Performance depends on threshold


Declare xn to be a positive if p(y=1|xn)>, otherwise declare it to be negative (y=0)
yn = 1 p(y = 1|xn ) >

Number of TPs and FPs depends on threshold . As we change , we get different (TPR, FPR) points.

T P R = p( = 1|y = 1) y F P R = p( = 1|y = 0) y

Example
i 1 2 3 4 5 6 7 8 9 yi 1 1 1 1 1 0 0 0 0 p(yi = 1|xi ) 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 yi ( = 0) 1 1 1 1 1 1 1 1 1 yi ( = 0.5) 1 1 1 1 1 0 0 0 0 yi ( = 1) 0 0 0 0 0 0 0 0 0

i 1 2 3 4 5 6 7 8 9

yi 1 1 1 1 1 0 0 0 0

p(yi = 1|xi ) 0.9 0.8 0.7 0.6 0.2 0.6 0.3 0.2 0.1

yi ( = 0) 1 1 1 1 1 1 1 1 1

yi ( = 0.5) 1 1 1 1 0 1 0 0 0

yi ( = 1) 0 0 0 0 0 0 0 0 0

Performance measures
EER- Equal error rate/ cross over error rate (false pos rate = false neg rate), smaller is better AUC - Area under curve, larger is better Accuracy = (TP+TN)/(P+N)

Precision-recall curves
Useful when notion of negative (and hence FPR) is not well defined, or too many negatives (rare event detection) Recall = of those that exist, how many did you find? Precision = of those that you found, how many correct? 2P R 2 = F = F-score is harmonic mean 1/P + 1/R R+P
prec = p(y = 1| = 1) y recall = p( = 1|y = 1) y

Word of caution
Consider binary classifiers A, B, C
1 0 A 1 0.9 0 . 0 0.1 0 B 1 0.8 0.1 . 0 0 0.1 C 1 0.78 0.12 . 0 0 0.1

Clearly A is useless, since it always predicts label 1, regardless of the input. Also, B is slightly better than C (less probability mass wasted on the offdiagonal entries). Yet here are the performance metrics.
Metric Accuracy Precision Recall F-score A 0.9 0.9 1.0 0.947 B 0.9 1.0 0.888 0.941 C 0.88 1.0 0.8667 0.9286

Mutual information is a better measure


The MI between estimated and true label is
1 1

I(Y , Y ) =
y =0

p(, y) y p(, y) log y p()p(y) y y=0

This gives the intuitively correct rankings B>C>A


Metric Accuracy Precision Recall F-score Mutual information A 0.9 0.9 1.0 0.947 0 B 0.9 1.0 0.888 0.941 0.1865 C 0.88 1.0 0.8667 0.9286 0.1735

You might also like