dev/notebooks/auto_examples/impute/plot_missing_values.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Imputing missing values before building an estimator\n\nMissing values can be replaced by the mean, the median or the most frequent\nvalue using the basic :class:`~sklearn.impute.SimpleImputer`.\n\nIn this example we will investigate different imputation techniques:\n\n- imputation by the constant value 0\n- imputation by the mean value of each feature combined with a missing-ness\n  indicator auxiliary variable\n- k nearest neighbor imputation\n- iterative imputation\n\nWe will use two datasets: Diabetes dataset which consists of 10 feature\nvariables collected from diabetes patients with an aim to predict disease\nprogression and California Housing dataset for which the target is the median\nhouse value for California districts.\n\nAs neither of these datasets have missing values, we will remove some\nvalues to create new versions with artificially missing data. The performance\nof\n:class:`~sklearn.ensemble.RandomForestRegressor` on the full original dataset\nis then compared the performance on the altered datasets with the artificially\nmissing values imputed using different techniques.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: Maria Telenczuk  <https://github.com/maikia>\n# License: BSD 3 clause"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Download the data and make missing values sets\n\n First we download the two datasets. Diabetes dataset is shipped with\n scikit-learn. It has 442 entries, each with 10 features. California Housing\n dataset is much larger with 20640 entries and 8 features. It needs to be\n downloaded. We will only use the first 400 entries for the sake of speeding\n up the calculations but feel free to use the whole dataset.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\n\nfrom sklearn.datasets import fetch_california_housing, load_diabetes\n\nrng = np.random.RandomState(42)\n\nX_diabetes, y_diabetes = load_diabetes(return_X_y=True)\nX_california, y_california = fetch_california_housing(return_X_y=True)\nX_california = X_california[:300]\ny_california = y_california[:300]\nX_diabetes = X_diabetes[:300]\ny_diabetes = y_diabetes[:300]\n\n\ndef add_missing_values(X_full, y_full):\n    n_samples, n_features = X_full.shape\n\n    # Add missing values in 75% of the lines\n    missing_rate = 0.75\n    n_missing_samples = int(n_samples * missing_rate)\n\n    missing_samples = np.zeros(n_samples, dtype=bool)\n    missing_samples[:n_missing_samples] = True\n\n    rng.shuffle(missing_samples)\n    missing_features = rng.randint(0, n_features, n_missing_samples)\n    X_missing = X_full.copy()\n    X_missing[missing_samples, missing_features] = np.nan\n    y_missing = y_full.copy()\n\n    return X_missing, y_missing\n\n\nX_miss_california, y_miss_california = add_missing_values(X_california, y_california)\n\nX_miss_diabetes, y_miss_diabetes = add_missing_values(X_diabetes, y_diabetes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Impute the missing data and score\nNow we will write a function which will score the results on the differently\nimputed data. Let's look at each imputer separately:\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "rng = np.random.RandomState(0)\n\nfrom sklearn.ensemble import RandomForestRegressor\n\n# To use the experimental IterativeImputer, we need to explicitly ask for it:\nfrom sklearn.experimental import enable_iterative_imputer  # noqa\nfrom sklearn.impute import IterativeImputer, KNNImputer, SimpleImputer\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.pipeline import make_pipeline\n\nN_SPLITS = 4\nregressor = RandomForestRegressor(random_state=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Missing information\nIn addition to imputing the missing values, the imputers have an\n`add_indicator` parameter that marks the values that were missing, which\nmight carry some information.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def get_scores_for_imputer(imputer, X_missing, y_missing):\n    estimator = make_pipeline(imputer, regressor)\n    impute_scores = cross_val_score(\n        estimator, X_missing, y_missing, scoring=\"neg_mean_squared_error\", cv=N_SPLITS\n    )\n    return impute_scores\n\n\nx_labels = []\n\nmses_california = np.zeros(5)\nstds_california = np.zeros(5)\nmses_diabetes = np.zeros(5)\nstds_diabetes = np.zeros(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Estimate the score\nFirst, we want to estimate the score on the original data:\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def get_full_score(X_full, y_full):\n    full_scores = cross_val_score(\n        regressor, X_full, y_full, scoring=\"neg_mean_squared_error\", cv=N_SPLITS\n    )\n    return full_scores.mean(), full_scores.std()\n\n\nmses_california[0], stds_california[0] = get_full_score(X_california, y_california)\nmses_diabetes[0], stds_diabetes[0] = get_full_score(X_diabetes, y_diabetes)\nx_labels.append(\"Full data\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Replace missing values by 0\n\nNow we will estimate the score on the data where the missing values are\nreplaced by 0:\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def get_impute_zero_score(X_missing, y_missing):\n    imputer = SimpleImputer(\n        missing_values=np.nan, add_indicator=True, strategy=\"constant\", fill_value=0\n    )\n    zero_impute_scores = get_scores_for_imputer(imputer, X_missing, y_missing)\n    return zero_impute_scores.mean(), zero_impute_scores.std()\n\n\nmses_california[1], stds_california[1] = get_impute_zero_score(\n    X_miss_california, y_miss_california\n)\nmses_diabetes[1], stds_diabetes[1] = get_impute_zero_score(\n    X_miss_diabetes, y_miss_diabetes\n)\nx_labels.append(\"Zero imputation\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### kNN-imputation of the missing values\n\n:class:`~sklearn.impute.KNNImputer` imputes missing values using the weighted\nor unweighted mean of the desired number of nearest neighbors.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def get_impute_knn_score(X_missing, y_missing):\n    imputer = KNNImputer(missing_values=np.nan, add_indicator=True)\n    knn_impute_scores = get_scores_for_imputer(imputer, X_missing, y_missing)\n    return knn_impute_scores.mean(), knn_impute_scores.std()\n\n\nmses_california[2], stds_california[2] = get_impute_knn_score(\n    X_miss_california, y_miss_california\n)\nmses_diabetes[2], stds_diabetes[2] = get_impute_knn_score(\n    X_miss_diabetes, y_miss_diabetes\n)\nx_labels.append(\"KNN Imputation\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Impute missing values with mean\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def get_impute_mean(X_missing, y_missing):\n    imputer = SimpleImputer(missing_values=np.nan, strategy=\"mean\", add_indicator=True)\n    mean_impute_scores = get_scores_for_imputer(imputer, X_missing, y_missing)\n    return mean_impute_scores.mean(), mean_impute_scores.std()\n\n\nmses_california[3], stds_california[3] = get_impute_mean(\n    X_miss_california, y_miss_california\n)\nmses_diabetes[3], stds_diabetes[3] = get_impute_mean(X_miss_diabetes, y_miss_diabetes)\nx_labels.append(\"Mean Imputation\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Iterative imputation of the missing values\n\nAnother option is the :class:`~sklearn.impute.IterativeImputer`. This uses\nround-robin linear regression, modeling each feature with missing values as a\nfunction of other features, in turn.\nThe version implemented assumes Gaussian (output) variables. If your features\nare obviously non-normal, consider transforming them to look more normal\nto potentially improve performance.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def get_impute_iterative(X_missing, y_missing):\n    imputer = IterativeImputer(\n        missing_values=np.nan,\n        add_indicator=True,\n        random_state=0,\n        n_nearest_features=3,\n        max_iter=1,\n        sample_posterior=True,\n    )\n    iterative_impute_scores = get_scores_for_imputer(imputer, X_missing, y_missing)\n    return iterative_impute_scores.mean(), iterative_impute_scores.std()\n\n\nmses_california[4], stds_california[4] = get_impute_iterative(\n    X_miss_california, y_miss_california\n)\nmses_diabetes[4], stds_diabetes[4] = get_impute_iterative(\n    X_miss_diabetes, y_miss_diabetes\n)\nx_labels.append(\"Iterative Imputation\")\n\nmses_diabetes = mses_diabetes * -1\nmses_california = mses_california * -1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Plot the results\n\nFinally we are going to visualize the score:\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n\nn_bars = len(mses_diabetes)\nxval = np.arange(n_bars)\n\ncolors = [\"r\", \"g\", \"b\", \"orange\", \"black\"]\n\n# plot diabetes results\nplt.figure(figsize=(12, 6))\nax1 = plt.subplot(121)\nfor j in xval:\n    ax1.barh(\n        j,\n        mses_diabetes[j],\n        xerr=stds_diabetes[j],\n        color=colors[j],\n        alpha=0.6,\n        align=\"center\",\n    )\n\nax1.set_title(\"Imputation Techniques with Diabetes Data\")\nax1.set_xlim(left=np.min(mses_diabetes) * 0.9, right=np.max(mses_diabetes) * 1.1)\nax1.set_yticks(xval)\nax1.set_xlabel(\"MSE\")\nax1.invert_yaxis()\nax1.set_yticklabels(x_labels)\n\n# plot california dataset results\nax2 = plt.subplot(122)\nfor j in xval:\n    ax2.barh(\n        j,\n        mses_california[j],\n        xerr=stds_california[j],\n        color=colors[j],\n        alpha=0.6,\n        align=\"center\",\n    )\n\nax2.set_title(\"Imputation Techniques with California Data\")\nax2.set_yticks(xval)\nax2.set_xlabel(\"MSE\")\nax2.invert_yaxis()\nax2.set_yticklabels([\"\"] * n_bars)\n\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can also try different techniques. For instance, the median is a more\nrobust estimator for data with high magnitude variables which could dominate\nresults (otherwise known as a 'long tail').\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.18"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}