dev/notebooks/auto_examples/classification/plot_classification_probability.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Plot classification probability\n\nThis example illustrates the use of\n:class:`sklearn.inspection.DecisionBoundaryDisplay` to plot the predicted class\nprobabilities of various classifiers in a 2D feature space, mostly for didactic\npurposes.\n\nThe first three columns shows the predicted probability for varying values of\nthe two features. Round markers represent the test data that was predicted to\nbelong to that class.\n\nIn the last column, all three classes are represented on each plot; the class\nwith the highest predicted probability at each point is plotted. The round\nmarkers show the test data and are colored by their true label.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Authors: The scikit-learn developers\nSPDX-License-Identifier: BSD-3-Clause\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import cm\n\nfrom sklearn import datasets\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom sklearn.gaussian_process import GaussianProcessClassifier\nfrom sklearn.gaussian_process.kernels import RBF\nfrom sklearn.inspection import DecisionBoundaryDisplay\nfrom sklearn.kernel_approximation import Nystroem\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import accuracy_score, log_loss, roc_auc_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import (\n    KBinsDiscretizer,\n    PolynomialFeatures,\n    SplineTransformer,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Data: 2D projection of the iris dataset\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "iris = datasets.load_iris()\nX = iris.data[:, 0:2]  # we only take the first two features for visualization\ny = iris.target\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.5, random_state=42\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Probabilistic classifiers\n\nWe will plot the decision boundaries of several classifiers that have a\n`predict_proba` method. This will allow us to visualize the uncertainty of\nthe classifier in regions where it is not certain of its prediction.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "classifiers = {\n    \"Logistic regression\\n(C=0.01)\": LogisticRegression(C=0.1),\n    \"Logistic regression\\n(C=1)\": LogisticRegression(C=100),\n    \"Gaussian Process\": GaussianProcessClassifier(kernel=1.0 * RBF([1.0, 1.0])),\n    \"Logistic regression\\n(RBF features)\": make_pipeline(\n        Nystroem(kernel=\"rbf\", gamma=5e-1, n_components=50, random_state=1),\n        LogisticRegression(C=10),\n    ),\n    \"Gradient Boosting\": HistGradientBoostingClassifier(),\n    \"Logistic regression\\n(binned features)\": make_pipeline(\n        KBinsDiscretizer(n_bins=5, quantile_method=\"averaged_inverted_cdf\"),\n        PolynomialFeatures(interaction_only=True),\n        LogisticRegression(C=10),\n    ),\n    \"Logistic regression\\n(spline features)\": make_pipeline(\n        SplineTransformer(n_knots=5),\n        PolynomialFeatures(interaction_only=True),\n        LogisticRegression(C=10),\n    ),\n}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Plotting the decision boundaries\n\nFor each classifier, we plot the per-class probabilities on the first three\ncolumns and the probabilities of the most likely class on the last column.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "n_classifiers = len(classifiers)\nscatter_kwargs = {\n    \"s\": 25,\n    \"marker\": \"o\",\n    \"linewidths\": 0.8,\n    \"edgecolor\": \"k\",\n    \"alpha\": 0.7,\n}\ny_unique = np.unique(y)\n\n# Ensure legend not cut off\nmpl.rcParams[\"savefig.bbox\"] = \"tight\"\nfig, axes = plt.subplots(\n    nrows=n_classifiers,\n    ncols=len(iris.target_names) + 1,\n    figsize=(4 * 2.2, n_classifiers * 2.2),\n)\nevaluation_results = []\nlevels = 100\nfor classifier_idx, (name, classifier) in enumerate(classifiers.items()):\n    y_pred = classifier.fit(X_train, y_train).predict(X_test)\n    y_pred_proba = classifier.predict_proba(X_test)\n    accuracy_test = accuracy_score(y_test, y_pred)\n    roc_auc_test = roc_auc_score(y_test, y_pred_proba, multi_class=\"ovr\")\n    log_loss_test = log_loss(y_test, y_pred_proba)\n    evaluation_results.append(\n        {\n            \"name\": name.replace(\"\\n\", \" \"),\n            \"accuracy\": accuracy_test,\n            \"roc_auc\": roc_auc_test,\n            \"log_loss\": log_loss_test,\n        }\n    )\n    for label in y_unique:\n        # plot the probability estimate provided by the classifier\n        disp = DecisionBoundaryDisplay.from_estimator(\n            classifier,\n            X_train,\n            response_method=\"predict_proba\",\n            class_of_interest=label,\n            ax=axes[classifier_idx, label],\n            vmin=0,\n            vmax=1,\n            cmap=\"Blues\",\n            levels=levels,\n        )\n        axes[classifier_idx, label].set_title(f\"Class {label}\")\n        # plot data predicted to belong to given class\n        mask_y_pred = y_pred == label\n        axes[classifier_idx, label].scatter(\n            X_test[mask_y_pred, 0], X_test[mask_y_pred, 1], c=\"w\", **scatter_kwargs\n        )\n\n        axes[classifier_idx, label].set(xticks=(), yticks=())\n    # add column that shows all classes by plotting class with max 'predict_proba'\n    max_class_disp = DecisionBoundaryDisplay.from_estimator(\n        classifier,\n        X_train,\n        response_method=\"predict_proba\",\n        class_of_interest=None,\n        ax=axes[classifier_idx, len(y_unique)],\n        vmin=0,\n        vmax=1,\n        levels=levels,\n    )\n    for label in y_unique:\n        mask_label = y_test == label\n        axes[classifier_idx, 3].scatter(\n            X_test[mask_label, 0],\n            X_test[mask_label, 1],\n            c=max_class_disp.multiclass_colors_[[label], :],\n            **scatter_kwargs,\n        )\n\n    axes[classifier_idx, 3].set(xticks=(), yticks=())\n    axes[classifier_idx, 3].set_title(\"Max class\")\n    axes[classifier_idx, 0].set_ylabel(name)\n\n# colorbar for single class plots\nax_single = fig.add_axes([0.15, 0.01, 0.5, 0.02])\nplt.title(\"Probability\")\n_ = plt.colorbar(\n    cm.ScalarMappable(norm=None, cmap=disp.surface_.cmap),\n    cax=ax_single,\n    orientation=\"horizontal\",\n)\n\n# colorbars for max probability class column\nmax_class_cmaps = [s.cmap for s in max_class_disp.surface_]\n\nfor label in y_unique:\n    ax_max = fig.add_axes([0.73, (0.06 - (label * 0.04)), 0.16, 0.015])\n    plt.title(f\"Probability class {label}\", fontsize=10)\n    _ = plt.colorbar(\n        cm.ScalarMappable(norm=None, cmap=max_class_cmaps[label]),\n        cax=ax_max,\n        orientation=\"horizontal\",\n    )\n    if label in (0, 1):\n        ax_max.set(xticks=(), yticks=())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Quantitative evaluation\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "pd.DataFrame(evaluation_results).round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Analysis\n\nThe two logistic regression models fitted on the original features display\nlinear decision boundaries as expected. For this particular problem, this\ndoes not seem to be detrimental as both models are competitive with the\nnon-linear models when quantitatively evaluated on the test set. We can\nobserve that the amount of regularization influences the model confidence:\nlighter colors for the strongly regularized model with a lower value of `C`.\nRegularization also impacts the orientation of decision boundary leading to\nslightly different ROC AUC.\n\nThe log-loss on the other hand evaluates both sharpness and calibration and\nas a result strongly favors the weakly regularized logistic-regression model,\nprobably because the strongly regularized model is under-confident. This\ncould be confirmed by looking at the calibration curve using\n:class:`sklearn.calibration.CalibrationDisplay`.\n\nThe logistic regression model with RBF features has a \"blobby\" decision\nboundary that is non-linear in the original feature space and is quite\nsimilar to the decision boundary of the Gaussian process classifier which is\nconfigured to use an RBF kernel.\n\nThe logistic regression model fitted on binned features with interactions has\na decision boundary that is non-linear in the original feature space and is\nquite similar to the decision boundary of the gradient boosting classifier:\nboth models favor axis-aligned decisions when extrapolating to unseen region\nof the feature space.\n\nThe logistic regression model fitted on spline features with interactions\nhas a similar axis-aligned extrapolation behavior but a smoother decision\nboundary in the dense region of the feature space than the two previous\nmodels.\n\nTo conclude, it is interesting to observe that feature engineering for\nlogistic regression models can be used to mimic some of the inductive bias of\nvarious non-linear models. However, for this particular dataset, using the\nraw features is enough to train a competitive model. This would not\nnecessarily the case for other datasets.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.17"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}