{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n# Receiver Operating Characteristic (ROC)\n\nExample of Receiver Operating Characteristic (ROC) metric to evaluate\nclassifier output quality.\n\nROC curves typically feature true positive rate on the Y axis, and false\npositive rate on the X axis. This means that the top left corner of the plot is\nthe \"ideal\" point - a false positive rate of zero, and a true positive rate of\none. This is not very realistic, but it does mean that a larger area under the\ncurve (AUC) is usually better.\n\nThe \"steepness\" of ROC curves is also important, since it is ideal to maximize\nthe true positive rate while minimizing the false positive rate.\n\nROC curves are typically used in binary classification to study the output of\na classifier. In order to extend ROC curve and ROC area to multi-label\nclassification, it is necessary to binarize the output. One ROC\ncurve can be drawn per label, but one can also draw a ROC curve by considering\neach element of the label indicator matrix as a binary prediction\n(micro-averaging).\n\nAnother evaluation measure for multi-label classification is\nmacro-averaging, which gives equal weight to the classification of each\nlabel.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>See also :func:`sklearn.metrics.roc_auc_score`,\n `sphx_glr_auto_examples_model_selection_plot_roc_crossval.py`</p></div>\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\nimport matplotlib.pyplot as plt\nfrom itertools import cycle\n\nfrom sklearn import svm, datasets\nfrom sklearn.metrics import roc_curve, auc\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import label_binarize\nfrom sklearn.multiclass import OneVsRestClassifier\nfrom sklearn.metrics import roc_auc_score\n\n# Import some data to play with\niris = datasets.load_iris()\nX = iris.data\ny = iris.target\n\n# Binarize the output\ny = label_binarize(y, classes=[0, 1, 2])\nn_classes = y.shape[1]\n\n# Add noisy features to make the problem harder\nrandom_state = np.random.RandomState(0)\nn_samples, n_features = X.shape\nX = np.c_[X, random_state.randn(n_samples, 200 * n_features)]\n\n# shuffle and split training and test sets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)\n\n# Learn to predict each class against the other\nclassifier = OneVsRestClassifier(\n svm.SVC(kernel=\"linear\", probability=True, random_state=random_state)\n)\ny_score = classifier.fit(X_train, y_train).decision_function(X_test)\n\n# Compute ROC curve and ROC area for each class\nfpr = dict()\ntpr = dict()\nroc_auc = dict()\nfor i in range(n_classes):\n fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])\n roc_auc[i] = auc(fpr[i], tpr[i])\n\n# Compute micro-average ROC curve and ROC area\nfpr[\"micro\"], tpr[\"micro\"], _ = roc_curve(y_test.ravel(), y_score.ravel())\nroc_auc[\"micro\"] = auc(fpr[\"micro\"], tpr[\"micro\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot of a ROC curve for a specific class\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"plt.figure()\nlw = 2\nplt.plot(\n fpr[2],\n tpr[2],\n color=\"darkorange\",\n lw=lw,\n label=\"ROC curve (area = %0.2f)\" % roc_auc[2],\n)\nplt.plot([0, 1], [0, 1], color=\"navy\", lw=lw, linestyle=\"--\")\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel(\"False Positive Rate\")\nplt.ylabel(\"True Positive Rate\")\nplt.title(\"Receiver operating characteristic example\")\nplt.legend(loc=\"lower right\")\nplt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plot ROC curves for the multiclass problem\nCompute macro-average ROC curve and ROC area\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# First aggregate all false positive rates\nall_fpr = np.unique(np.concatenate([fpr[i] for i in range(n_classes)]))\n\n# Then interpolate all ROC curves at this points\nmean_tpr = np.zeros_like(all_fpr)\nfor i in range(n_classes):\n mean_tpr += np.interp(all_fpr, fpr[i], tpr[i])\n\n# Finally average it and compute AUC\nmean_tpr /= n_classes\n\nfpr[\"macro\"] = all_fpr\ntpr[\"macro\"] = mean_tpr\nroc_auc[\"macro\"] = auc(fpr[\"macro\"], tpr[\"macro\"])\n\n# Plot all ROC curves\nplt.figure()\nplt.plot(\n fpr[\"micro\"],\n tpr[\"micro\"],\n label=\"micro-average ROC curve (area = {0:0.2f})\".format(roc_auc[\"micro\"]),\n color=\"deeppink\",\n linestyle=\":\",\n linewidth=4,\n)\n\nplt.plot(\n fpr[\"macro\"],\n tpr[\"macro\"],\n label=\"macro-average ROC curve (area = {0:0.2f})\".format(roc_auc[\"macro\"]),\n color=\"navy\",\n linestyle=\":\",\n linewidth=4,\n)\n\ncolors = cycle([\"aqua\", \"darkorange\", \"cornflowerblue\"])\nfor i, color in zip(range(n_classes), colors):\n plt.plot(\n fpr[i],\n tpr[i],\n color=color,\n lw=lw,\n label=\"ROC curve of class {0} (area = {1:0.2f})\".format(i, roc_auc[i]),\n )\n\nplt.plot([0, 1], [0, 1], \"k--\", lw=lw)\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel(\"False Positive Rate\")\nplt.ylabel(\"True Positive Rate\")\nplt.title(\"Some extension of Receiver operating characteristic to multiclass\")\nplt.legend(loc=\"lower right\")\nplt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Area under ROC for the multiclass problem\nThe :func:`sklearn.metrics.roc_auc_score` function can be used for\nmulti-class classification. The multi-class One-vs-One scheme compares every\nunique pairwise combination of classes. In this section, we calculate the AUC\nusing the OvR and OvO schemes. We report a macro average, and a\nprevalence-weighted average.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"y_prob = classifier.predict_proba(X_test)\n\nmacro_roc_auc_ovo = roc_auc_score(y_test, y_prob, multi_class=\"ovo\", average=\"macro\")\nweighted_roc_auc_ovo = roc_auc_score(\n y_test, y_prob, multi_class=\"ovo\", average=\"weighted\"\n)\nmacro_roc_auc_ovr = roc_auc_score(y_test, y_prob, multi_class=\"ovr\", average=\"macro\")\nweighted_roc_auc_ovr = roc_auc_score(\n y_test, y_prob, multi_class=\"ovr\", average=\"weighted\"\n)\nprint(\n \"One-vs-One ROC AUC scores:\\n{:.6f} (macro),\\n{:.6f} \"\n \"(weighted by prevalence)\".format(macro_roc_auc_ovo, weighted_roc_auc_ovo)\n)\nprint(\n \"One-vs-Rest ROC AUC scores:\\n{:.6f} (macro),\\n{:.6f} \"\n \"(weighted by prevalence)\".format(macro_roc_auc_ovr, weighted_roc_auc_ovr)\n)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
}
},
"nbformat": 4,
"nbformat_minor": 0
}