{

"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# <img style=\"float: left; padding-right: 10px; width: 45px\"
src=\"https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styl
es/iacs.png\"> CS109A Introduction to Data Science\n",
"\n",
"## Standard Section 5: Logistic Regression and PCA\n",
"\n",
"**Harvard University**<br/>\n",
"**Fall 2018**<br/>\n",
"**Section Leaders:** Mehul Smriti Raje, Ken Arnold, Karan Motwani, Cecilia
Garraffo<br/>\n",
"**Instructors:** Pavlos Protopapas, Kevin Rader"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [

"<style>h1{padding-top:25px;padding-bottom:25px;text-align:left;padding-left:10px;ba
ckground-color:#DDDDDD;color:black}h2{padding-top:10px;padding-bottom:10px;text-alig
n:left;padding-left:5px;background-color:#EEEEEE;color:black}div.exercise{background
-color:#ffcccc;border-color:#E9967A;border-left:5px solid
#800080;padding:0.5em}div.theme{background-color:#DDDDDD;border-color:#E9967A;border
-left:5px solid
#800080;padding:0.5em;font-size:18pt}p.q1{padding-top:5px;padding-bottom:5px;text-al
ign:left;padding-left:5px;background-color:#EEEEEE;color:black}header{padding-top:35
px;padding-bottom:35px;text-align:left;padding-left:10px;background-color:#DDDDDD;co
lor:black}</style>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#RUN THIS CELL \n",
 "from IPython.core.display import HTML\n",

"HTML('<style>h1{padding-top:25px;padding-bottom:25px;text-align:left;padding-left:1
0px;background-color:#DDDDDD;color:black}h2{padding-top:10px;padding-bottom:10px;tex
t-align:left;padding-left:5px;background-color:#EEEEEE;color:black}div.exercise{back
ground-color:#ffcccc;border-color:#E9967A;border-left:5px solid
#800080;padding:0.5em}div.theme{background-color:#DDDDDD;border-color:#E9967A;border
-left:5px solid
#800080;padding:0.5em;font-size:18pt}p.q1{padding-top:5px;padding-bottom:5px;text-al
ign:left;padding-left:5px;background-color:#EEEEEE;color:black}header{padding-top:35
px;padding-bottom:35px;text-align:left;padding-left:10px;background-color:#DDDDDD;co
lor:black}</style>')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this section we will be covering Logistic Regression and PCA using the Wine
dataset. The goal is to get you familiarized with the implementation of Logistic
Regresison and PCA through the use of a dataset we haven't seen. The exercises below
will help you be able to answer parts of Homework 5.\n",
"\n",
"Specifically, we will: \n",
" \n",
" 1. Import data, explore with analyse relationship with response.\n",
" 2. Fit a linear regression model for classification, understand drawbacks
and interpret results.\n",
" 3. Fit a logistic regression model for classification, compare performance
and interpret results.\n",
" 3. Use PCA to reduce dimensionality, fit model to PCA components and
compare performance.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For this section we will be using the following packages:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
 "/anaconda3/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56:
FutureWarning: The pandas.core.datetools module is deprecated and will be removed in
a future version. Please use the pandas.tseries module instead.\n",
" from pandas.core import datetools\n"
]
}
],
"source": [
"# Data and Stats packages\n",
"import numpy as np\n",
"import pandas as pd\n",
"import statsmodels.api as sm\n",
"from statsmodels.api import OLS\n",
"from sklearn import metrics, datasets\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.decomposition import PCA\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.linear_model import LogisticRegressionCV\n",
"from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
"from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis\n",
"from sklearn.preprocessing import PolynomialFeatures\n",
"from sklearn.neighbors import KNeighborsClassifier\n",
"from sklearn.model_selection import cross_val_score\n",
"from sklearn.metrics import accuracy_score\n",
"from sklearn.model_selection import KFold\n",
"from sklearn.preprocessing import MinMaxScaler\n",
"\n",
"# Visualization packages\n",
"%matplotlib inline\n",
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"matplotlib.rcParams['figure.figsize'] = (13.0, 6.0)\n",
"\n",
"# Other\n",
"import itertools\n",
"import tqdm\n",
"\n",
"# Aesthetic settings\n",
"from IPython.display import display\n",
"pd.set_option('display.max_columns', 999)\n",
"pd.set_option('display.width', 500)\n",
"sns.set_style('whitegrid')\n",
"sns.set_context('talk')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
 "# Part One"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import Dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" <th>species</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>7.0</td>\n",
" <td>3.2</td>\n",
" <td>4.7</td>\n",
" <td>1.4</td>\n",
 " <td>versicolor</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>6.4</td>\n",
" <td>3.2</td>\n",
" <td>4.5</td>\n",
" <td>1.5</td>\n",
" <td>versicolor</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6.9</td>\n",
" <td>3.1</td>\n",
" <td>4.9</td>\n",
" <td>1.5</td>\n",
" <td>versicolor</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>5.5</td>\n",
" <td>2.3</td>\n",
" <td>4.0</td>\n",
" <td>1.3</td>\n",
" <td>versicolor</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>6.5</td>\n",
" <td>2.8</td>\n",
" <td>4.6</td>\n",
" <td>1.5</td>\n",
" <td>versicolor</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width petal_length petal_width species\n",
"0 7.0 3.2 4.7 1.4 versicolor\n",
"1 6.4 3.2 4.5 1.5 versicolor\n",
"2 6.9 3.1 4.9 1.5 versicolor\n",
"3 5.5 2.3 4.0 1.3 versicolor\n",
"4 6.5 2.8 4.6 1.5 versicolor"
]
},
"metadata": {},
"output_type": "display_data"
}
 ],
"source": [
"#Import dataframe\n",
"flowers_df = pd.read_csv('flowers.csv')\n",
"del flowers_df['Unnamed: 0']\n",
"display(flowers_df.head())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see, `species` here is the response variable which defines flower
type and the 4 predictor variables are flower characteristics. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>100.000000</td>\n",
" <td>100.000000</td>\n",
" <td>100.000000</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>6.262000</td>\n",
" <td>2.872000</td>\n",
" <td>4.906000</td>\n",
" <td>1.676000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.662834</td>\n",
" <td>0.332751</td>\n",
" <td>0.825578</td>\n",
" <td>0.424769</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>4.900000</td>\n",
" <td>2.000000</td>\n",
" <td>3.000000</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>5.800000</td>\n",
" <td>2.700000</td>\n",
" <td>4.375000</td>\n",
" <td>1.300000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>6.300000</td>\n",
" <td>2.900000</td>\n",
" <td>4.900000</td>\n",
" <td>1.600000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>6.700000</td>\n",
" <td>3.025000</td>\n",
" <td>5.525000</td>\n",
" <td>2.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
 " <td>7.900000</td>\n",
" <td>3.800000</td>\n",
" <td>6.900000</td>\n",
" <td>2.500000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width petal_length petal_width\n",
"count 100.000000 100.000000 100.000000 100.000000\n",
"mean 6.262000 2.872000 4.906000 1.676000\n",
"std 0.662834 0.332751 0.825578 0.424769\n",
"min 4.900000 2.000000 3.000000 1.000000\n",
"25% 5.800000 2.700000 4.375000 1.300000\n",
"50% 6.300000 2.900000 4.900000 1.600000\n",
"75% 6.700000 3.025000 5.525000 2.000000\n",
"max 7.900000 3.800000 6.900000 2.500000"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#Describe dataset\n",
"flowers_df['species'] = flowers_df['species'] == 'versicolor'\n",
"display(flowers_df.describe())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Explore Data "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check which variables have high correlations and distinctive patterns with the
response. Any patterns worth mentioning?"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
 "outputs": [
{
"data": {

"text/plain": [
"<matplotlib.figure.Figure at 0x105379198>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"sns.pairplot(flowers_df, hue=\"species\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### This plot is great because it not only shows correlations but also
relationship with the response. The left diagonal shows the histogram for each
predictor with the response subset. You can re-create the left diagonal with the
code below."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {

"text/plain": [
"<matplotlib.figure.Figure at 0x1113498d0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#Explore dataset for relationships with Quality\n",
"subset_species_0, subset_species_1 = flowers_df[flowers_df['species'] == 0],
flowers_df[flowers_df['species'] == 1]\n",
"\n",
 "#Get set of predictors\n",
"columns = list(flowers_df.columns)\n",
"columns.remove('species')\n",
"\n",
"#Plot relationship\n",
"fig, ax = plt.subplots(2, 2, figsize=(15, 15))\n",
"alpha = 0.4\n",
"count = 0\n",
"for i in range(2):\n",
" for j in range(2):\n",
" ax[i, j].hist(subset_species_0[columns[count]], alpha=alpha,
label='Species = 0', normed=True)\n",
" ax[i, j].hist(subset_species_1[columns[count]], alpha=alpha,
label='Species = 1', normed=True)\n",
" ax[i, j].set_title(columns[count]+' vs Quality')\n",
" ax[i, j].set_xlabel(columns[count])\n",
" ax[i, j].set_ylabel('Frequency')\n",
" ax[i, j].legend()\n",
" count += 1\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can observe that `petal_length`, `petal_width` are useful distinguishing
predictors for flower species."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Split and Normalize Data"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Length of Training Set : 64\n",
"Length of Testing Set : 36\n"
]
}
 ],
"source": [
"#Split data into train and test\n",
"np.random.seed(1001)\n",
"msk = np.random.rand(len(flowers_df)) < 0.7\n",
"data_train = flowers_df[msk]\n",
"data_test = flowers_df[~msk]\n",
"\n",
"#Split predictor and response columns\n",
"x_train, y_train = data_train.drop(['species'], axis=1),
data_train['species']\n",
"x_test, y_test = data_test.drop(['species'], axis=1), data_test['species']\n",
"\n",
"print(\"Length of Training Set :\",len(data_train))\n",
"print(\"Length of Testing Set :\",len(data_test))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What is the right way to normalize?"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#Function to normalize data \n",
"def normalize(df, train_dict, flag=True):\n",
" for i in df.columns:\n",
" if flag:\n",
" train_dict[i] = (df[i].min(), df[i].max())\n",
" df[i] = (df[i] - train_dict[i][0]) / (train_dict[i][1] -
train_dict[i][0])\n",
" return df, train_dict\n",
"\n",
"#Normalize train and get characteristics of training columns\n",
"x_train, train_dict = normalize(x_train, {}, True)\n",
"x_test, train_dict = normalize(x_test, train_dict, False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
 "Peeking at any information from the test set is cheating, we use the entire
dataset for atmost data exploration. No information, including min, max, mean std,
should be used from the test set or under the influence of the test set. Thus, the
right way to normalize is to store the characteristics of the training set and apply
it to both datasets independently."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training Dataframe :\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>64.000000</td>\n",
" <td>64.000000</td>\n",
" <td>64.000000</td>\n",
" <td>64.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>0.442187</td>\n",
" <td>0.416992</td>\n",
" <td>0.466747</td>\n",
" <td>0.414583</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.223429</td>\n",
" <td>0.195451</td>\n",
" <td>0.213806</td>\n",
" <td>0.285936</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>0.266667</td>\n",
" <td>0.312500</td>\n",
" <td>0.307692</td>\n",
" <td>0.200000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>0.466667</td>\n",
" <td>0.437500</td>\n",
" <td>0.435897</td>\n",
" <td>0.333333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>0.575000</td>\n",
" <td>0.500000</td>\n",
" <td>0.602564</td>\n",
" <td>0.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>1.000000</td>\n",
" <td>1.000000</td>\n",
 " <td>1.000000</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width petal_length petal_width\n",
"count 64.000000 64.000000 64.000000 64.000000\n",
"mean 0.442187 0.416992 0.466747 0.414583\n",
"std 0.223429 0.195451 0.213806 0.285936\n",
"min 0.000000 0.000000 0.000000 0.000000\n",
"25% 0.266667 0.312500 0.307692 0.200000\n",
"50% 0.466667 0.437500 0.435897 0.333333\n",
"75% 0.575000 0.500000 0.602564 0.666667\n",
"max 1.000000 1.000000 1.000000 1.000000"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"Testing Dataframe :\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>36.000000</td>\n",
" <td>36.000000</td>\n",
" <td>36.000000</td>\n",
" <td>36.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>0.475000</td>\n",
" <td>0.425347</td>\n",
" <td>0.527778</td>\n",
" <td>0.514815</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.217981</td>\n",
" <td>0.231368</td>\n",
" <td>0.204979</td>\n",
" <td>0.270267</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>0.000000</td>\n",
" <td>-0.125000</td>\n",
" <td>0.128205</td>\n",
" <td>0.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>0.358333</td>\n",
" <td>0.296875</td>\n",
" <td>0.384615</td>\n",
" <td>0.316667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>0.450000</td>\n",
" <td>0.437500</td>\n",
" <td>0.525641</td>\n",
" <td>0.533333</td>\n",
 " </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>0.600000</td>\n",
" <td>0.562500</td>\n",
" <td>0.666667</td>\n",
" <td>0.733333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>0.933333</td>\n",
" <td>1.000000</td>\n",
" <td>0.948718</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width petal_length petal_width\n",
"count 36.000000 36.000000 36.000000 36.000000\n",
"mean 0.475000 0.425347 0.527778 0.514815\n",
"std 0.217981 0.231368 0.204979 0.270267\n",
"min 0.000000 -0.125000 0.128205 0.000000\n",
"25% 0.358333 0.296875 0.384615 0.316667\n",
"50% 0.450000 0.437500 0.525641 0.533333\n",
"75% 0.600000 0.562500 0.666667 0.733333\n",
"max 0.933333 1.000000 0.948718 1.000000"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"print(\"Training Dataframe :\")\n",
"display(x_train.describe())\n",
"print(\"\\n\\nTesting Dataframe :\")\n",
"display(x_test.describe())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some maximum values in Testing Dataframe are not 1, some minimum values are >
0. \n",
"Is this wrong? What could be an explanation for this? Could we see negative
values in test or values greater than 1?"
 ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Part Two"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Perform Classification using Multiple Linear Regression"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training Set Accuracy: 98.4375%\n",
"Testing Set Accuracy: 94.44444444444444%\n"
]
}
],
"source": [
"#Add constant to x_train and x_test\n",
"x_train_cst = sm.add_constant(x_train)\n",
"x_test_cst = sm.add_constant(x_test)\n",
"\n",
"#Training\n",
"model = OLS(y_train, x_train_cst).fit()\n",
"\n",
"#Predict\n",
"y_pred_train = model.predict(x_train_cst)>0.5\n",
"y_pred_test = model.predict(x_test_cst)>0.5\n",
"\n",
"#Perfromance Evaluation\n",
"train_score = accuracy_score(y_train, y_pred_train)*100\n",
"test_score = accuracy_score(y_test, y_pred_test)*100\n",
"\n",
"print(\"Training Set Accuracy:\",str(train_score)+'%')\n",
"print(\"Testing Set Accuracy:\",str(test_score)+'%')"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Significant Predictors :\n",
"\n",
"const 1.302238e-21\n",
"petal_width 1.368759e-06\n",
"dtype: float64 \n",
"\n",
"Confidence Intervals:\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0.5%</th>\n",
" <th>99.5%</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>const</th>\n",
" <td>1.021095</td>\n",
" <td>1.466538</td>\n",
" </tr>\n",
" <tr>\n",
 " <th>sepal_length</th>\n",
" <td>-0.493553</td>\n",
" <td>1.000970</td>\n",
" </tr>\n",
" <tr>\n",
" <th>sepal_width</th>\n",
" <td>-0.115400</td>\n",
" <td>0.921418</td>\n",
" </tr>\n",
" <tr>\n",
" <th>petal_length</th>\n",
" <td>-2.049886</td>\n",
" <td>0.164881</td>\n",
" </tr>\n",
" <tr>\n",
" <th>petal_width</th>\n",
" <td>-1.768611</td>\n",
" <td>-0.597113</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0.5% 99.5%\n",
"const 1.021095 1.466538\n",
"sepal_length -0.493553 1.000970\n",
"sepal_width -0.115400 0.921418\n",
"petal_length -2.049886 0.164881\n",
"petal_width -1.768611 -0.597113"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#Get significant predictors and Confidence Intervals using Stats Model
Summary\n",
"print(\"Significant Predictors :\\n\",)\n",
"print(model.pvalues[model.pvalues<0.01],\"\\n\")\n",
"\n",
"thresh = 0.01\n",
"intervals = model.conf_int(alpha=thresh)\n",
"intervals = intervals.rename(index=str,
columns={0:str(thresh/2*100)+\"%\",1:str((1-thresh/2)*100)+\"%\"})\n",
"print(\"Confidence Intervals:\",)\n",
"display(intervals)"
]
},
 {
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, we have used a higher probability threshold (p<0.01) since we have very
few datapoints. Does this make sense?\n",
"\n",
"`petal_width` seems like the only significant predictor, is this intuitive
given the histogram above?"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>Predicted</th>\n",
" <th>False</th>\n",
" <th>True</th>\n",
" <th>All</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Actual</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
 " <th>False</th>\n",
" <td>22</td>\n",
" <td>2</td>\n",
" <td>24</td>\n",
" </tr>\n",
" <tr>\n",
" <th>True</th>\n",
" <td>0</td>\n",
" <td>12</td>\n",
" <td>12</td>\n",
" </tr>\n",
" <tr>\n",
" <th>All</th>\n",
" <td>22</td>\n",
" <td>14</td>\n",
" <td>36</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Predicted False True All\n",
"Actual \n",
"False 22 2 24\n",
"True 0 12 12\n",
"All 22 14 36"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Get Performance by Class (Lookup Confusion Matrix)\n",
"pd.crosstab(y_test, y_pred_test, margins=True, rownames=['Actual'],
colnames=['Predicted'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Can you calculate True Positive Rate? False Positive Rate? \n",
"\n",
"If this was a medical test for cancer, what do we want ideally, Low False
Positive rate or Low False Negative rate? What are the implications?"
]
},
 {
"cell_type": "markdown",
"metadata": {},
"source": [
"# Part Three"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Perform Classification using Multiple Logistic Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [

"[https://towardsdatascience.com/logistic-regression-detailed-overview-46c4da4303bc]
"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training Set Accuracy: 100.0%\n",
"Testing Set Accuracy: 86.11111111111111%\n"
]
}
],
"source": [
"#Add constant to x_train and x_test\n",
"x_train_cst = sm.add_constant(x_train)\n",
"x_test_cst = sm.add_constant(x_test)\n",
"\n",
"#Training\n",
"model = LogisticRegression(C=100000).fit(x_train_cst, y_train)\n",
"\n",
"#Predict\n",
"y_pred_train = model.predict(x_train_cst)\n",
"y_pred_test = model.predict(x_test_cst)\n",
"\n",
"#Perfromance Evaluation\n",
 "train_score = accuracy_score(y_train, y_pred_train)*100\n",
"test_score = accuracy_score(y_test, y_pred_test)*100\n",
"\n",
"print(\"Training Set Accuracy:\",str(train_score)+'%')\n",
"print(\"Testing Set Accuracy:\",str(test_score)+'%')"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>Predicted</th>\n",
" <th>False</th>\n",
" <th>True</th>\n",
" <th>All</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Actual</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>False</th>\n",
" <td>20</td>\n",
" <td>4</td>\n",
" <td>24</td>\n",
 " </tr>\n",
" <tr>\n",
" <th>True</th>\n",
" <td>1</td>\n",
" <td>11</td>\n",
" <td>12</td>\n",
" </tr>\n",
" <tr>\n",
" <th>All</th>\n",
" <td>21</td>\n",
" <td>15</td>\n",
" <td>36</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Predicted False True All\n",
"Actual \n",
"False 20 4 24\n",
"True 1 11 12\n",
"All 21 15 36"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Get Performance by Class \n",
"pd.crosstab(y_test, y_pred_test, margins=True, rownames=['Actual'],
colnames=['Predicted'])"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#Helper function for visualization\n",
"def visualize_prob(model, x, y, ax):\n",
" \"\"\"\n",
" Function Name : visualize_prob\n",
" Description : A function to visualize the probabilities predicted by a
Logistic Regression model\n",
 " Input: \n",
" model (Logistic regression model)\n",
" x (n x d array of predictors in training data)\n",
" y (n x 1 array of response variable vals in training data: 0 or 1)\n",
" ax (an axis object to generate the plot)\n",
" \"\"\"\n",
" \n",
" #Use the model to predict probabilities for x\n",
" y_pred = model.predict_proba(x)\n",
" \n",
" #Separate the predictions on the label 1 and label 0 points\n",
" ypos = y_pred[y==1]\n",
" yneg = y_pred[y==0]\n",
" \n",
" #Count the number of label 1 and label 0 points\n",
" npos = ypos.shape[0]\n",
" nneg = yneg.shape[0]\n",
" \n",
" #Plot the probabilities on a vertical line at x = 0, \n",
" #with the positive points in blue and negative points in red\n",
" pos_handle = ax.plot(np.zeros((npos,1)), ypos[:,1], 'bo', label = 'Species
1', alpha=0.2)\n",
" neg_handle = ax.plot(np.zeros((nneg,1)), yneg[:,1], 'ro', label = 'Species
0', alpha=0.2)\n",
"\n",
" #Line to mark prob 0.5\n",
" ax.axhline(y = 0.5, color = 'k', linestyle = '--')\n",
" \n",
" #Add y-label and legend, do not display x-axis, set y-axis limit\n",
" ax.set_ylabel('Probability of Species Class')\n",
" ax.legend(loc = 'best')\n",
" ax.get_xaxis().set_visible(False)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"image/png":
"iVBORw0KGgoAAAANSUhEUgAAAtgAAAFlCAYAAADRb58zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgA
ACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGx
vdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XmYFNW5+PEvzAADooCsimtcjoiSkEEMXo1r8rtu6NVrjEmMRIm
KW1yiRqLXuGByQ2KIqKgkGGNcosaoGI2aqAm5istoXBCOqOCGiKCC7LP9/qieYRiYnpqhe7odvp/n6YfuqtN
Vb7fjO++cOudUh9raWiRJkiTlRsdCByBJkiS1JxbYkiRJUg5ZYEuSJEk5ZIEtSZIk5ZAFtiRJkpRDFtiSJEl
SDpUWOgC1HyGE3wEnNNNs/xjjk608/k+AU2OMA1K23w94AhgUY5zVmnOmPM+TwL4NNlUDS4BngMtjjE+38Hh
bA1+NMd6WsyDTnfcQ4P0Y40tteV5JhZfv/N3oXJsAJ8YYJ2ZejwYmA51ijFUbevws530PGNhgUxXwCfAkcEm
MMbbweLsB28cYp+YsyObP2QE4Hng0xji/rc6rlrMHW7n0A2CLBo+PgF822vbUBhz/F8DuLWj/VOacszfgnGn
dz5rP+AXgUJIi+8kQwkEtPNbvgcNzG152IYRtgb+QxC9p45Pv/N3Qj4ALG7y+Ddgin8V1AxNY83l2Ar4B9AO
eDiEMbuGxHgb2zG14zToAuAXo3sbnVQvZg62ciTEuBhbXvQ4h1ABLc/VXdoxxKbC0Be1XA231F/7KRp/znRD
CdOCvwI0hhNCCXx4dch9eUZ5TUpHId/5uZK18E2NcAazIw3nWZ1mjzzQ3hHAw8DwwkaSATctcrSZZYKvNhRB
GAVeS9FqcDMwE/gMYAVwG7AF0A94DJsUYx2fe9xMaDBEJIdQCpwD/RTJE4yPgT8AFMcaqxkNEMkM5KoBNgGO
AWuCfwJgY44eZY24O/JqkB7oj8EeSnoLKGOOolnzOGGNtCOGXwCOZzzYthNAZ+AlJr8k2wEqSXqEzY4yzGw4
3CSEcG2Ps0Nx7Mm2/nvlOdyP5RfV34JwY4/uZ/ZsBP898V92AV0guif49hLAdMCcT9sMhhFta+lklbTxCCLs
C44GvkuSbacAPY4xzMvv7AdcB+5Hkz1eBy2KMD4YQrgR+nGlXC+wD7EKDISKZoRzXAMOAg4HPSDorzogxLs+
8d3uSgnhfYDlwU+b5ozHGK1vyeWKMK0II1wHXhRC2jDHOCyH0BH5G8rtgAMkVyb9lYviowXCTH4cQvhlj3LG
592TiPp6k935H4FNgKnBujPGzzP6tSK7W/j+S31HPkfxOeylzNfSxTNizQwiXtPSzqu04RESFMhDYlSSBjiF
JRo8CERhOUijeDfw8hJDtEtwvgLuAL5Ik23OAb2VpfwawDNgLOJGkt+KnUD+2bSowFDiCJPH3A45rzQfM+Hf
m3y9m/v0ZMIrkM+9EUvDuRPLLBOAo4GnWDDlp9j2ZPwruJ/kFNJgkMe8MTGnwuR4m+U6PAsrr2md6bt4l+c4
h+e5+sAGfV1I7likAp5HkjT2BrwGrgWdDCHXzY24gyfFfI8lJ/wL+FEIYSJLPJgAfkOS4Z5o41eXAdODLwMU
k447PzsTQDXicpLNgX5IhdQcCe2/AR6vL1UMy//6e5PfEsSTF8EmZz/M/mf1DM59hAvCVNO8JIXyRZHjHBCB
k2h1M8p0QQugO/APo3ODzzASmZ8Z7/5OkowWSTptfbcDnVZ7Zg61CuiLG+CbU90ZcAfyybihFCOES4AKShNd
UEr4txnhz5vkvQggnkfSG/76J9m/FGM/LPI8hhDtYc0lwH5Lk+OUY44uZGL4FvNnaD0gygQagR+bf54G/xBj
/nnn9dgjhTpICmhjjxyGE1aw95CTre0h6tctIhsO8HWOcE0I4Buid2X9A5nNtHWN8L7Ptf0MIXybpGXk4hPB
RXbyZS8WStD6nAwtijKfWbcj0yr5PUlCOI/kDfx4wJ8a4OITwI5IOgM9ijEtDCMuAmrocF0JY33keizFenXk
+O4RwHEluB/gmSafMnjHGBZlj/DdrrsS1RuNc/QjwVN3vApK8exyZAjzTi11DMuRkYZr3kHSMdCCZTP52Zv+
hQE1m/7eBvsA3M0McAc4OIewD/CDG+P0QQl2cC2OMyzbg8yrPLLBVSPWTDzNF4WTg1BDC7sAOrElKJVmO0Xh
1kMUkf/23pv0wYEWD5Fh36fDZLMdrTl2y/jRzvNtDCPuGEK4i6eEIJD35i5o6QHPviTH+O4TwB5JLsleEEP4
OPATckTnElzP/zmr0i6wza36pSFIaXwZ2CiE0ng/TlSQvAVxK0lP7UWYuyqPAH2KMS1pwnvXl6rp8OgyYXVd
cA2SGdWzIhPa1cjUwCTg0hPBtkt9Hg0j+cMg20bO59/yFZNjiQyGEeSTDPaYC92b2f5lkSM3HjXJ1F5Khgfo
ccYiICql+UksIYRDJ8JBjgLkkwx++lOIYq9azLdskkGztq4AOmSEVuVKe+fcFgMw4vwdIxoE/CowmGcvYpDT
viTEeT9I78hOgE3A98H8hhDKS/89XkXyfDR+7subSpiSl0ZFkqELjfBKA8wBijH8CtiTpkZ0BfB94JdMTm1Z
zuTrX9Us5yZjnf4cQOgIPkgx1qQH+TPK76a6m3pzmPTHGFTHGA0i+r1+TfEd3ZtpC8plms+53O4hkOIk+R+z
BVrEYQzKRZb8YYy1ACKGuB7utZk2/SDLU4kuZ52QmGJaTTBpsjbNIEub0EEJv4DTgezHG39U1CCFcxNqfsbb
Bvmbfk/meTgPOyqwrOzGEcCDJ5JohJBMauwA9YowVDY7xM5JfYpc2PKckZfEKyRCN9zOrf5D5Q/4O4A8hhD+
TjCl+KMZ4N3B3CKETyVC7Y0jGb29ovnkRODmE0LfB5MG+JL3GLZbJ82OAR2KMH4YQhpOMjd4nxvivTJsOJJM
xGw7LaPg5hjX3nhDC/sChMcYfAi+RzDE6BbghM5fmFeA7JENpPmwQ3y0kc3NuwFz9uWGBrWLxDsmEl5EhhBd
JJsbUjb/r0hYBxBinhRD+CdwSQjgd+Bi4CNiK5pNaWYMJPp1IxkWfCxwEfD2zokjdMlgjQwhPk/RWnEAyabH
huOfPgO1Csjb1+ynes5AkKXcOIfw8E+t3SYZ/zCJJ7i8At4cQzgTeIJnMeAHJxKG6cwLsHkJ4LsbY5JAVSRu
1a0muov0xs7LTKpL5MwcC58cYa0IIu5DkrFNIrkjuQzJmum6oxGdAj0y7ua2I4XaSiYN3hhDOz2wbT9JB0ly
u3qRBru5M0vN+CUlv8lGZ7fNIbhh2bGYox2YkEyy/SLISVZ3PSIbLbJnyPUtJxlQvJxlCswlwNEme/oRk7tA
FwJ8zn2sBye+R40hWWak7J8DQEMJHzpkpXg4RUbGYSJJcfkMya3o8cCPJpci2XMj/mMz5/wL8H0kRO531X65
s6AiSGeUfkBSwt5EsHbVnjPEfAJnJm0cB25PMWH+cJLmfQvLLpm7Q3bXAdpk4+jX3nhjjPOAQkiEiz5Ak84H
AQTHGJTHGauDrJD1HfyC5ZPsN4Pi6u0XGGD8mGT94BVA3aVSS1hJjfItkeb5OJCtePEVSTB4QY3wj02xUZvu
dwOskq4CcE2O8M7P/jySdKi+R5K6WxrCKZLWkKpIVSv6aiWUezefqs1mTq2eR5L1XgWExxtczx3+PpPPh68B
rJOOkyXyOQSGETTOv65bT+3fm3FnfE2N8jmSox2GZz/5PkqL7kBhjbYzxU5I/Rt4nGW7yIskV1cPqesUz53q
AJJf/JNUXpoLoUFvr1QYJ6odj7E1ymXBlg+1vADfHGMcVLDhJEgAhWbt/1xjjQw22lZH0Ao+KMf6xULFJdRw
iIq2xmqRX4NYQwoTMttEkvcFNTm6RJLWpMuCBEML/kIz97k5y85bFJEvlSQXnEBEpI3MnrcNIbsjyPMlQiz2
Ar9XdMVGSVFgxxlkkEy2PJhneMY1kmMr+mWEWUsE5RESSJEnKIXuwJUmSpByywJYkSZJyqN1NcqyoqHDMi6T
PtfLy8ra6uVLBmbMlfd6tL2e3uwIboLy8vPlGklSEKioqmm/UzpizJX1eNZWzHSIiSZIk5ZAFtiRJkpRDFti
SJElSDllgS5IkSTlkgS1JkiTlkAW2JEmSlEMW2JIkSVIOWWBLLfDXvxY6AklSSyyYV1XoELQRapc3mpFyafJ
kOPnkdbffdBN8//ttH48kKbs5cTW/HPUyq2e9RUnlSqo7ldF5ly9w3u+GsH3oXOjwtBEoSIEdQhgO3Bdj3LK
J/ccB44B+wJPASTHGD9suQinRVHENa7a35yK7pgY6ep1ro2fO1ufJnLiaq/b5C72XvcsmtZ9RQg3Vqzqy7OX
5XLXPu4yddmi7LrLN28WhTf8ThBA6hBBOBB4F1vvTHUIYAtwAHAf0BeYD17dZkFIDTRXXafd/HlVXw/vvw0s
vwYsvJv++/36yPR/mzJnDmDFj2GOPPRg6dCgjR47k7rvvzs/JMm644QbOP//8vBz7scce4+ijj87LsduaOVu
fR1ef8CJbLXudzWoXU0INHaihhBo2q13MVste5+oTXix0iDnXlnnbnJ1OW/dgjwW+QdLTcWETbb4N3B9jfAY
ghHAh8GEIoV+McUGak6xcuTIXsUpASbMtVq7MU+VZANXV8PrrHVixYs22Vatg2TJYsAB23rmWkua/ktRqamo
46aSTOOKII/jpT39K586deeGFFzjnnHPo2rUrBx10UO5O1sCoUaOA3OaKqqoqbr31Vq6//np23HHH9pKHzNn
63Cl77Tk61y6je+1SutYupyO11NCBFR26sZTulL32HCtXfrHQYeZMW+Ztc3Z6bV1gTwGuAvbN0mYX4Om6FzH
GRSGETzPbUyXrGTNmbEiMElA3oXFIs+1++csZ/Od/5j2cNrFgQScWLerU5P758yvp168yZ+dbvHgx77//Pjv
ttBNvvvkmAF27duXYY49lzpw5zJgxg3vuuYd58+axZMkS3nzzTbbaaitOOukktttuOwBmzpzJbbfdxvz589l
iiy044YQT2HHHHQF49913ufnmm5k7dy6bbropxxxzDHvvvTf33HMP7733HmeffTY1NTXcd999PPnkk6xevZq
hQ4dy/PHH061bN5YtW8YNN9zArFmz6Nq1K7vtthujRo2ic+d1O3MnT57MBx98wMEHH8zLL7/cXvKQOVufKwv
mVbHZ6oX0ql1IaW2Sq2oy+7ryGZ06rKSqpiNPPPYS/bZsH9PQ2jJvm7PTa9OfrhjjBwAhhGzNNgGWN9q2HOi
W9jyDBw9ucWxSY4MHw8UXN9/uvPPaz89bdXUHevRoen9pKQweXJvTcw4bNoxf/vKXHHrooQwbNozddtttrf+
H//nPf/LnP/+Zn/3sZxx44IH89re/ZeLEiUydOpWFCxdy9dVXc+WVV/LVr36Vxx9/nHHjxvHAAw/QrVs3zjv
vPI488khOOukkYoyMHj2ar3/96/Tr148lS5YwePBgfve73/HKK69w2223semmm3LZZZdx//33c+WVV3Lttdf
Ss2dP/vGPf7BixQpGjx7NnDlzOOqoo9b5HGPHjqVfv37cf//9zJ49u9V5qJiKTXO2Pm8GD6rh9ZI/0bm6Gjq
sOwq2M9VsVrKc/Q/cvd0MVG7rvG3OXltTObsY/3xbDnRttK0bsDTtAcrKynIakJRNWVnTPQefJzU1UFJCs5c
SO3fO7e+lKVOmcMcdd/Doo48yZcoUAL7+9a9zySWX0KtXL0pLSxkxYgRHHnkkAGeeeSZ33HEHM2bM4OWXX2b
PPffkkEMOAagfC/jkk0+y7bbbsmLFCs466yw6duzIsGHDuOOOO9h6660pLS2lpKSEsrIy7rvvPs4777z63pU
LL7yQgw46iHHjxtGrVy9mzZrF3/72N/bZZx/uu+8+Ojbx4bfZZhsAOnXqRIcOHTamPGTOVlHZZIte8N5nWfe
XdUv9919RK0TeNmenU4wF9kygvrskhNAH2DyzXWozNTUwdixcdVXTbcaObT8ztjt2THo6qrIsGVtamvvP2qV
LF0aNGsWoUaNYtWoVFRUVjB8/nrFjxzJp0iRgTSIEKCkpoW/fvixcuJB58+Yxbdo0hg0bVr+/qqqK8vJyunf
vTr9+/dZKroMGDVrn/B988AEXXHABJQ1+Q5WWljJv3jxGjRrF6tWrmTJlCmPHjqW8vJwrr7yyPrELMGermNT
UcOj5g3nw/I8oWb1ind3Vnbty2PmD203iLkTeNmenU4w/XXcAR4cQ9g4hlAE/BR6OMS4qcFzayHTsCEcfDSe
dtP79J52U7G8HObpenz4btr+lHnroIQ444ABqa5PLl126dGGvvfZizJgxzJo1q77dggVrhvJWVVWxYMECBgw
YQN++fTnkkEN4/vnn6x9Tp05l9OjR9O/fnwULFlBTU1P/3ttuu42XXnpprRj69u3L9ddfX//+p59+mvvvv59
tttmG2bNnc8QRRzB16lSefPJJevfuzRVXXJHbL+Hzz5yt4tGxI1vssRUjLxtGx622oKYk6UesKSml41ZbMPK
yYWyxx1btKnG3Zd42Z6dXFD9hIYQbQgg3AMQY/w18n2RyzQJgS+B7BQxPG7GaGthvP7j11uRx/vlrnu+3X7K
/PRkwALo2vtif0bVrsj+XRowYwfLlyxk3bhyLFi2itraWt99+mzvvvJP999+/vt20adN46qmnqKys5LrrrqN
Xr14MHTqUQw89lCeeeIKnn36a2tpaKioqGDlyJK+88gpDhgyhR48eTJ48maqqKl5++WUmTJhA9+7d14rhyCO
P5LrrrmPBggVUVlYyYcIERo8eTW1tLXfddReXXnopS5cupVevXpSVldGzZ8/cfgmfQ+ZsFbX+/en/HzsyauI
ejP7NXhz9868w+jd7MWriHvT/jx2hf/9CR5hTbZm3zdnpFWSISIzxSaBPg9enNtp/F3BXG4clraNjR+jSJVn
yCGBIg0VFunRpV50gQDKOLwSYPx8WLkwuO5aWJj0gAwY0P86vpXr16sXtt9/OhAkTOOyww1i+fDm9e/fm8MM
P5/TTT69vN2TIECZPnswZZ5zB4MGDufHGGykpKWG77bZjwoQJjB8/nrlz57L55ptz0UUXMWLECAAmTZrE5Zd
fzuTJk+nduzfjxo1jhx12WCuGU045hcrKSo499liWLFnCrrvuyk033URpaSnnnHMOl1xyCQceeCCVlZUMHz6
cK6+8MrdfwueAOVufKwMGwOLFScLq148eOzYYDpKPnoICa8u8bc5Or0NdN397UVFRUVteXl7oMNQO1NQkC/Z
XV8OiRfDpp8nzkhLo2RN6906eDx3a/grtOsUwTHHixInMnj2ba665prCBtJGKigrKy8s7FDqOtmLOVl5UV7d
dT0GRKXTeNmcninGSo1QU6iaPAPTrlzwaJ658TPorJu35s0lqx0pKYODA5FHoirONbUQftaj5n0HKovHkkMa
JK9eT/iRJOWbFqQKwB1vKom4o34p1V3tqj0P5itKZZ55Z6BAkSSmZsxP+WSdlUTd5ZMCANcNFSkuT1yG0+6F
8kiSpFezBlpqxEQ/lkyRJrWCpILWAxbUkSWqO5YIkSZKUQxbYkprW3m5VKUntnXm7KDgGW9LaNuIbNEjS55J
5u+jYgy1pjepqiDFJ1FVVybaqquR1jMn+HJszZw5jxoxhjz32YOjQoYwcOZK777475+dp6IYbbuD888/P6TG
feuopDjvsML70pS/xrW99izlz5uT0+JK0Xm2ct83Z6VhgS1pj/vz1L/oNyfb583N6upqaGkaPHs1uu+3GtGn
TqKio4OKLL2b8+PE88sgjOT1XQ6eeeirjx4/P2fEWLlzIGWecwbnnnsuzzz7LXnvtxXnnnZez40tSk9owb5u
z07PAlrTGwoUbtr+FPvnkE9577z1GjhxJWVkZHTt2ZPjw4Zx//vlUVlYCMHHiRM455xy++93v8qUvfYljjjm
GmTNn1h/jueee4+ijj2bYsGEcc8wxvPzyy/X7Xn/9db7zne8wdOhQDjzwQB544IH6Y5511lkAVFdXc+2113L
AAQcwYsQILrroIpYuXQrAkiVLOO200xg+fDj7778/P/7xj1m1atU6n+PRRx9l0KBBHHDAAXTu3JkxY8bw7rv
v8uqrr+b0+5KkdbRh3jZnp2eBLSlRU7Pm8mJTqqpyOoGmd+/eDB8+nO9973tcc801TJ8+neXLl3PMMcdw2GG
H1bd7+OGH+eY3v8lzzz3Hvvvuy2mnncbq1auZN28ep5xyCmPGjGH69OmceOKJfP/73+fTTz9l9erVnHLKKXz
lK1/hmWee4Ve/+hWXXnopb7755lox3HzzzTz22GPcdtttPPbYY6xcuZIrrrgCgClTplBSUsK//vUv7rvvPmb
MmFGf8Bt666232GGHHepfl5SUsPXWW/PGG2/k7LuSpHW0cd42Z6dngS0p0bHjmttVNqW0NOeLgf/mN7/hO9/
5DtOnT2f06NEMHz6cc889l08++aS+zYgRIzjkkEPo1KkTY8aMYfny5bzwwgs8+OCD7Lnnnhx00EGUlpZy8ME
Hs/POO/PII4/wwgsvsHz5ck477TQ6d+7MkCFDuP322+nfv/9a57/nnns444wz2GKLLejevTs//OEPeeCBB1i
1ahWbbropM2bM4C9/+QuVlZXce++9HHPMMet8hhUrVtC1a9e1tnXt2pUVTV22laRcKEDeNmen4yoiktbo0yf
7eL0+fXJ+yi5dujBq1ChGjRrFqlWrqKioYPz48YwdO5ZJkyYBsM0229S3LykpoW/fvixcuJB58+Yxbdo0hg0
bVr+/qqqK8vJyunfvTr9+/ejY4BfLoEGD1jn/Bx98wAUXXEBJg5n2paWlzJs3j1GjRrF69WqmTJnC2LFjKS8
v58orr2S77bZb6xhdu3Zl5cqVa21bsWIF3bp126DvRpKa1cZ525ydjj3YktYYMAAa/VVfr2vXZH8OPfTQQxx
wwAHU1tYCSeLea6+9GDNmDLNmzapvt2DBgvrnVVVVLFiwgAEDBtC3b18OOeQQnn/++frH1KlTGT16NP3792f
BggXUNLg0etttt/HSSy+tFUPfvn25/vrr69//9NNPc//997PNNtswe/ZsjjjiCKZOncqTTz5J79696y9FNvS
FL3xhrRno1dXVvPPOO+y44445+64kab3aMG+bs9OzwJa0RkkJhJAk5LrLjqWlyesQcr6e6ogRI1i+fDnjxo1
j0aJF1NbW8vbbb3PnnXey//7717ebNm0aTz31FJWVlVx33XX06tWLoUOHcuihh/LEE0/w9NNPU1tbS0VFBSN
HjuSVV15hyJAh9OjRg8mTJ1NVVcXLL7/MhAkT6N69+1oxHHnkkVx33XUsWLCAyspKJkyYwOjRo6mtreWuu+7
i0ksvZenSpfTq1YuysjJ69uy5zuf42te+xquvvsqjjz7K6tWrmTRpEgMGDGDXXXfN6fclSetow7xtzk7PISK
S1lZSAgMHJo+ampyPuW6oV69e3H777UyYMIHDDjuM5cuX07t3bw4//HBOP/30+nZDhgxh8uTJnHHGGQwePJg
bb7yRkpIStttuOyZMmMD48eOZO3cum2++ORdddBEjRowAYNKkSVx++eVMnjyZ3r17M27cuLUmtgCccsopVFZ
Wcuyxx7JkyRJ23XVXbrrpJkpLSznnnHO45JJLOPDAA6msrGT48OFceeWV63yOuh6Vq666igsvvJBBgwYxceJ
EOnTokLfvTpLqtVHeNmen16Gum7+9qKioqC0vLy90GJJyZOLEicyePZtrrrmm0KG0iYqKCsrLyzeaytycLbU
v5uyEQ0QkSZKkHLLAliRJknLIMdiSitqZZ55Z6BAkSSmZsxP2YEuSJEk5ZIEtSZIk5ZAFtiRJkpRDFtiSJEl
SDllgS5IkSTlkgS1JkiTlkAW2JEmSlEMW2JIkSVIOWWBLkiRJOWSBLUmSJOWQBbYkSZKUQxbYkiRJUg5ZYEu
SJEk5ZIEtSZIk5ZAFtiRJkpRDFtiSJElSDllgS5IkSTlkgS1JkiTlkAW2JEmSlEMW2JIkSVIOWWBLkiRJOWS
BLUmSJOWQBbYkSZKUQxbYkiRJUg5ZYEuSJEk5ZIEtSZIk5VBpW54shDAUuBEYDMwGTo0xTl9Pu4uBMUA34Fl
gTIzxrbaMVZI2duZsSWqdNuvBDiGUAVOBm4GewDXAvSGEzo3aHQ6cAAwD+gFvAL9pqzglSeZsSdoQbTlEZH+
gJsY4KcZYGWOcAiwCDm/UbudMXCVAB6AaWNGGcUqSzNmS1GptOURkF+C1RtsiyaXHPzXYdidwCvAuSaKeB/x
HS060cuXK1kcpSQJztiS1WlsW2JsAyxttW04yZq+hLsC/gEOBD4BfAX8MIfxHjLE2zYlmzJixgaFK0kbPnC1
JrdSWBfZyoGujbd2ApY22XQPcG2OcDRBCOAv4DNgNeCXNiQYPHrxhkUpSgRRRsWnOlqRmNJWzW1VghxA2JZn
QMiPGuCDl22YCZzQ+FHB7o23bkPSI1KkGaoDKtPGVlZWlbSpJ7Z45W5LaVqoCO4SwK3Ar8APgVeA5YAdgRQh
hZIzx7ykO8zjQJYRwJnADcDzQH3ikUbu/AOeHEP5KMpbvp5lzvp4mVkna2JmzJamw0q4iMhF4E5gFnAh0B7Y
AxgE/T3OAGOMq4GDgOOBj4ExgZIxxWQjh4RDC2EzTn5BMoPkXSbLeATgyxliTMlZJ2tiZsyWpgDrU1jY/ByW
EsAzYLcY4J4TwD2BmjPHUEMK2wKwYY+NxegVTUVFRW15eXugwJKlVKioqKC8v77AhxzBnS1LbaCpnp+3B/gz
oHULoC+wFPJTZPgj4KDchSpJyxJwtSQWUdpLjn4B7SGaVvw/8NYTwbeDXwKQ8xSZJah1ztiQVUNoe7LOACcC
jwEExxtUkd+36n8xDklQ8zNmSVECperBjjNUkyRqAEEIHYBrwXtobCUiS2oY5W5IKK+0yfQNJbiYwjuTWuY8
DewILQgiHxBhfzF+IkqSWMGdLUmGlHSKosJb6AAAfdUlEQVRyHbA5sBA4gWSizF4k4/x+nZ/QJEmtZM6WpAJ
KW2AfAJwaY3wHOBKYGmN8Bria5O5gkqTiYc6WpAJKW2BXAqUhhO7AfqxZ8mlLYHEe4pIktZ45W5IKKO0yfY8
Ct5CsrboceDCE8J/AtcCf8xSbJKl1zNmSVEBpe7BPJrkN7mLg0BjjUpIxffcC5+YpNklS65izJamA0i7T9xl
wdqNtv8pLRJKkDWLOlqTCSrtM32YkNy4YTHKzAoAOQBfgizHGbfMTniSppczZklRYaYeITCZJ1rXAfwGrgW2
Aw4Df5Cc0SVIrmbMlqYDSFthfB74dY/wWMBP4RYxxT2AisFu+gpMktYo5W5IKKG2B3RWYlXn+GrBH5vn1wFd
zHZQkaYOYsyWpgNIW2LOBEZnnM0luuQtJEt8k10FJkjaIOVuSCijtOti/AH4fQugE3AG8HEIoAYYD/8xXcJK
kVjFnS1IBperBjjHeAhwIzIwxvk4yUWYTkkR9Yv7CkyS1lDlbkgorbQ82Mcb/a/D8b8Df8hKRJGmDmbMlqXC
aLLBDCM+RLPHUrBjj8JxFJElqMXO2JBWPbD3YD7ZZFJKkDWXOlqQi0WSBHWO8rC0DkSS1njlbkopH1jHYIYR
tgP8BLosxvttg+41AZ+DHMcZ5+Q1RkpSGOVuSikOTq4iEEL4ATAf2Bno02v0sybqqz2YSuiSpgMzZklQ8si3
TdwXwAjAkxvhqwx0xxt8C5cDrwOX5C0+SlJI5W5KKRLYCe3+Sy4yr17czxrgC+AnJWquSpMIyZ0tSkchWYG8
KLGrm/e8CPXMXjiSplczZklQkshXYM4ERzbx/BDA3Z9FIklrLnC1JRSJbgf0b4KchhB3WtzOEsCPwM+D3+Qh
MktQi5mxJKhLZ1sG+KYSwL/BqCOHPwHPAYqAXsAdwBPAQcHVbBCpJapo5W5KKR9Z1sGOM3w4hHAt8D7iQJFE
vJEnc344x3pv/ECVJaZizJak4ZC2wAWKMfwT+2AaxSJI2kDlbkgov2xhsSZIkSS1kgS1JkiTlkAW2JEmSlEM
W2JIkSVIONTvJsU4IYXfgvRjjJyGEw4H/Bp4Hro0x1uYrQElSy5mzJalwUvVghxBOBl4EhoQQhgL3AP2AscA
V+QtPktRS5mxJKqy0Q0R+CIyKMf4DGAW8FGM8GPgmyXqrkqTiYc6WpAJKW2BvDfwj8/wwYGrm+TtAj1wHJUn
aIOZsSSqgtGOw3wQOCyG8B2zPmmQ9GngtH4FJklrNnC1JBZS2wP4f4I5M+9tijP8OIfwaOBE4Il/BSZJaxZw
tSQWUaohIjPFeYCugPMZ4fGbzJOALMcbH8xWcJKnlzNmSVFipl+kDlgBfCyH8F/BrYADwSV6ikiRtKHO2JBV
I2mX6tgdmAv9LssxTT+B0YEZmCShJUpEwZ0tSYaVdReQa4DFgW2BlZttxwF+AX+UhLklS65mzJamA0hbYewN
Xxxhr6jbEGKuAK4HyfAQmSWo1c7YkFVDaAnsVsPl6tn8BWJq7cCRJOWDOlqQCSltg3wpcG0LYM/O6bwhhJHA
jcHteIpMktZY5W5IKKO0qIhcBVwFPAl2Ap4Eq4PrMvlQyk2tuBAYDs4FTY4zT19Puv4CfAgOBGcApMcaX0p5
HkjZy5mxJKqC062BXxRgvAHoBuwNDgV4xxrNjjKvTHCOEUEZyN7GbSWa0XwPcG0Lo3KjdUGAK8H2SW/r+Gbg
73ceRJJmzJamwmuzBDiEcAjwWY6zMPG9sYAgBgBjjQynOtT9QE2OclHk9JYRwDnA48KcG7U4BJscYp2XiuBr
4WwihY8MJO5KkNczZklQ8sg0ReZDkxgQLMs+bUguUpDjXLsBrjbZFkkuPDZP1l4EHQwiPA0OAF4HTW5KoV65
c2XwjSWpfzNmSVCSaLLBjjB3X93wDbAIsb7RtOdCt0bbNgTEkvSSvAJcBD4QQdsssM9WsGTNmbGCokvT5Ys6
WpOKR+lbpIYTjgBUxxvsyr38LPBRj/FP2d9ZbDnRttK0b6y4ZtQq4N8b4fOY8lwDnkvSmvJrmRIMHD04ZkiQ
Vl1wVm+ZsScq/pnJ2qgI7hHA+cAlwWoPN80nG5PWPMV6f4jAzgTMaH5p1l4yKJBNq6nRo8EilrKwsbVNJanf
M2ZJUWGl7sE8HvtlwYkyM8cchhGdIbrubJlk/DnQJIZwJ3AAcD/QHHmnU7nfA7SGE3wPPkNx5bDYpe0IkSeZ
sSSqktOP0+gBvrmf7LGCLNAeIMa4CDgaOAz4GzgRGxhiXhRAeDiGMzbR7ADgVmJxptydwRIyxNmWskrSxM2d
LUgF1qK1tPgeGEP4GvAOcXDdpJYTQEbgOGBxj/Gpeo2yBioqK2vLy8kKHIUmtUlFRQXl5eerhFetjzpakttF
Uzk47RORc4O/A2yGElzPbds+8f33rrUqSCsecLUkFlPZOji+TzAi/iuSy40yScXY7xRhfyF94kqSWMmdLUmG
lXqYvxrgohPBnkhsPPANsGmP8LG+RSZJazZwtSYWTdpm+TYApwDFADbAzcHUIoTdwVIzxo/yFKElqCXO2JBV
W2lVEfg4MJLnkuCKz7SKgjGTJJ0lS8TBnS1IBpS2wjwDOiTG+XrchxjiTZGmm/8xHYJKkVjNnS1IBpS2wNwO
WrGd7DdApd+FIknLAnC1JBZS2wH4UuCSEUDdmuzaE0Bf4BfBYXiKTJLWWOVuSCihtgX0msCOwCOgG1N3EoCf
wg/yEJklqJXO2JBVQqlVEYowfAF8JIewP7Jp530zgMW+HK0nFxZwtSYWVeh3sjCXAUqAa+MBELUlFzZwtSQW
Qdh3sLYH7gGHAx0AJsFkI4Qng6Bjj4vyFKElqCXO2JBVW2jHYk0nWUt0xxtgnxtgLGESypuq1+QpOktQq5mx
JKqC0Bfa+wOkxxrfqNmTWVz0DGJmPwCRJrWbOlqQCSltgvwPstJ7tfYEPcxeOJCkHzNmSVEBpJzlOAG4IIew
E/AuoAoYClwC/CyEcUtcwxvhQzqOUJLWEOVuSCihtgX1D5t+frWff2AbPa0km00iSCsecLUkFlHYd7LRDSSR
JBWbOlqTCauk62IQQOgG7AwtijO/lPiRJUq6YsyWp7WXt5QghjAkhvBxC2DbzegjwJvA88HYI4bYQQuc2iFO
S1AxztiQVhyYL7BDCicDVwF+BupsS/IGk13sPkjVVdwIuynOMkqRmmLMlqXhkGyJyGnBWjHEyQAhhGLAbcGG
MsSKz7VJgInBZvgOVJGVlzpakIpGtwB4E/L3B66+RzDh/sMG2mcBWeYhrg7z55ptN7uvTpw89evSofz1nzhx
qamqabL/55pvTq1ev+tdz586lurq6yfY9e/akd+/e9a/fffddVq9e3WT7zTbbjL59+9a/fv/991m5cmWT7bt
3707//v3rX3/wwQcsX768yfbdunVjiy22qH/94YcfsnTp0ibbl5WVMXDgwPrXH330EUuWLGmyfefOndl6663
rXy9atIhPP/20yfalpaVsu+229a8/+eQTPv744ybbd+zYke23377+9eLFi1m4cGGT7QF22GGH+uefffYZCxY
syNp+++23p2PH5GLOsmXLmD9/ftb22267LaWlyf86K1asYN68eVnbb7311nTunFyVX7VqFe+9l30Y7MCBAyk
rKwOgsrKSd955J2v7LbbYgm7dugFQXV3N3Llzs7bv378/3bt3B6C2tpa33nora/u+ffuy2Wab1b9+6623qK2
tbbJ979696dmzZ/3r5v6f6dWrF5tvvnn963feeYfKysom2/fo0YM+ffrUv37vvfdYtWpVk+033XRT+vXrV/9
63rx5rFixosn2m2yyCQMGDKh/PX/+fJYtW9Zk+65du7LlllvWv16wYAGfffZZk+27dOnCVlutSZsLFy5k8eK
c3LncnI0525y9LnO2ObuhNsvZtbW1633svPPOn+y8886DGrz++8477/xOozZf2XnnnT9s6hiFeDz//PO1JL9
U1vu49tpraxvabLPNsrYfN27cWu232mqrrO0vvPDCtdoPHjw4a/tTTz11rfZ77bVX1vbHHXfcWu0PPvjgrO0
PPfTQtdofe+yxWdvvvffea7U/+eSTs7bffffd12p//vnnZ22/zTbbrNX+iiuuyNq+Z8+ea7W/5pprsrbv0KH
DWu1vueWWrO2B2iVLltS3v++++5pt/+6779a3f/zxx5ttP2PGjPr2FRUVzbZ/6qmn6tu/8cYbzbZ/+OGH69t
/9NFHzbb/4x//WN9+1apVzba/6aab1vpOy8rKsrYfP378Wu379++ftf3FF1+8Vvuddtopa/uzzjprrfbDhg3
L2v6EE05Yq/2BBx6Ytf2RRx65Vvujjjoqa/sDDjhgrfajRo3K2r68vHyt9j/4wQ/W2v/888/X1rYi95mzk4c
525zd+GHONmc3fLRVzs42yfFfwIkAIYRdgK8Cf2rU5ixgepZjSJLahjlbkopEh9omLhuEEL4EPAEsBAYAnwL
DYowfhhAOBs4G9gH2qRvfVwwqKipqG17qaMzLjV5ubMzLjV5ubKjQlxs//fRTysvLOzR5gCaYsxPmbHN2Y+Z
sc3ZDbZWzmyywAUIIWwBHAzXAH2OMizLbLwRGAONijM81eYACqKioqC0vLy90GJLUKhUVFa0qsMGcLUltram
cnfVGMzHGD4Br17P9f3MYmyQpB8zZklQcvJ2uJEmSlEMW2JIkSVIOWWBLkiRJOZTtVun/DCH0zzz/bgihS9u
FJUlqCXO2JBWPbD3YdbfZBbgZ2CxLW0lSYZmzJalIZFtF5M/AYyGEWqADMD+EsN6GMcaSPMQmSUrPnC1JRSJ
bgf0d4DqgFzAV+C7JjQskScXHnC1JRaLJAjvGWAs8BRBC+B5wT4yx6VvvSJIKxpwtScUj641m6sQYbwkhjAg
h/AgYRDJ2exbw6xjjY/kMUJLUMuZsSSqsVMv0hRD+G5gGrAAmktwpbAXwUAhhZP7CkyS1lDlbkgorVQ828BP
gRzHGXzTYNiGE8MPMvgdyHJckqfV+gjlbkgom7Y1mvgDct57t95FcfpQkFQ9ztiQVUNoC+01g3/Vs3x94L3f
hSJJywJwtSQWUdojIz4GbQgi7Ac8CtcAI4PvAD/IUmySpdczZklRAaVcRuTVzw4IfAKeQTJaZBRwXY7w/f+F
JklrKnC1JhZW2B5sY463ArXmMRZKUI+ZsSSqctGOwJUmSJKVggS1JkiTlkAW2JEmSlENp7+S4Z74DkSTlhjl
bkgor7STHJ0MI7wO3AbfHGGMeY5IkbRhztiQVUNoCuz9wNPBN4KIQwkskifvOGOP8tCcLIQwFbgQGA7OBU2O
M07O0PxH4eYyxT9pzSJLM2ZJUSKmGiMQYl8QYb44x/j9gIPBb4GvA7BDCoyGE40MIXbMdI4RQBkwFbgZ6Atc
A94YQOjfR/gvA1ek/iiQJzNmSVGitmeS4GdAL6AN0AiqBc4B3QgiHZnnf/kBNjHFSjLEyxjgFWAQc3rhhCKE
E+D1wUyvikyStYc6WpDaWaohICGEb4BsklxuHAtNJkumdMcZFmTZXAlNILk2uzy7Aa422RZJLj39qtP1HwAz
gIeDENDE2tHLlypa+RZLaDXO2JBVW2jHYc4HXgduBb8QY31pPm2kkibwpmwDLG21bDnRruCGEUA4cDwzLPFp
sxowZrXmbJLUXczFnS1LBpC2wvwI8H2OsabgxhFAKDI0xPhdjfAR4JMsxlgONx/x1A5Y2OF5X4BZgdIxxaQg
hZXhrGzx4cKveJ0mFlqNi05wtSW2gqZydtsB+GhgAfNRo+w7AP2jUo9GEmcAZjbYFkh6WOsOALwAPZhJ1KdA
thPApMCTG+E6aYMvKytI0k6T2ypwtSQXUZIEdQhgDXJZ52QF4LYRQ26hZd+DVlOd6HOgSQjgTuIHkkmJ/GvS
gxBin0SDxhxD2A+5xySdJys6cLUnFI1sP9mRgGclKI1OAK4DFDfbXklwqfDzNiWKMq0IIB5Mk6quAN4CRMcZ
lIYSHgWkxxqta/hEkSZizJalodKitbdzBsa4Qwr7A/8UYq/If0oapqKioLS8vL3QYktQqFRUVlJeXd9iQY5i
zJaltNJWzsw0R+TlwWYxxGXAocGhTE1hijBfkKlBJUsuZsyWpeGQbIrIHyU0J6p43pfkucElSvpmzJalINFl
gxxj3X99zSVLxMWdLUvHINkTkkJTHqI0xPpyjeCRJrWDOlqTikW2IyIMpj1ELlOQgFklS65mzJalIZBsi0rE
tA5EktZ45W5KKR7YhIrsCs2KMNZnnTamNMc7MfWiSpLTM2ZJUPLINEXmV5Fa7CzLPa0nuDtaYlxslqfDM2ZJ
UJLIV2NsDHzV4LkkqXuZsSSoS2cZgv934eQhhc2AXYDUwO8a4uIm3S5LakDlbkopHth7seiGETYCbgG+w5tJ
iZQhhMvDDGOPKPMUnSWohc7YkFVbaWefXA0OArwE9gF7AyMzrX+cnNElSK5mzJamAUvVgA0cD+8UYn2+w7ZE
QwonAQ8ApOY9MktRa5mxJKqC0PdgfAZs0se+zHMUiScoNc7YkFVBz62DXmQT8LoRwHvAMUA18EbgGuCKvEUq
SmmXOlqTi0dw62I3XUb0ns40G268Hbsx9aJKkFjBnS1KRaG4dbEnS54M5W5KKRKp1sOuEELqwZsmnDkAXYCi
wTltJUtsxZ0tS8Ui7DvZ+wG9Yfw9JJVCWw5gkSRvAnC1JhZV2FZGrgdeAQ4BlwFHAmcAnwHfzE5okqZXM2ZJ
UQGkL7F2BH8UYHwFeAFbEGK8HzgB+mK/gJEmtYs6WpAJKW2CvBFZlnkfgS5nnTwG75DooSdIGMWdLUgGlLbD
/BYwNIWwKPA/8VwihE/BVYGm+gpMktYo5W5IKKG2BfR6wN3Ay8AdgU2AJcAtwbX5CkyS1kjlbkgoo1SoiMcY
IhBBCtxjjihDCcGA/YFGM8Zl8BihJahlztiQVVqoCGyCEUAp8LYQwGFgNzASezVdgkqTWM2dLUuGkXQd7B+D
vwObAbJKhJTsBb4UQjogxzslfiJKkljBnS1JhpR2DfTMwHRgYYyyPMQ4FtgbmAjfkKTZJUuuYsyWpgNIW2Hs
Al8YYP6vbEGP8BLgI2CcfgUmSWs2cLUkFlLbAfgn4ynq2DyFZY1WSVDzM2ZJUQE2OwQ4hnNbg5XRgUgihHHg
GqAa+SHJXsPF5jVCS1CxztiQVj2yTHM9v9PpD4PDMo85CYBRweW7DkiS1kDlbkopEkwV2jHH7tgxEktR65mx
JKh4tWQd7M+AkYBDJ2O2ZwK0xxgV5ik2S1ErmbEkqnFSTHEMIQ0jWUj0H6A30Ac4GXgsh7Jq/8CRJLWXOlqT
CStuDPQF4BDgpxlgJEELoBEwBrgb+Mz/hSZJawZwtSQWUdpm+rwBX1SVqgMzznwL/kY/AJEmtZs6WpAJKW2A
vALZaz/atgGW5C0eSlAPmbEkqoLRDRP4ATA4hnA08C9QCI0guNd6ep9gkSa1jzpakAkpbYF8ObAHcw5pe7yr
geuBHeYhLktR65mxJKqC0BfYewBjgXCAAK4E3YozL8xWYJKnVzNmSVEBpC+z7gYNijP8mudwoSSpe5mxJKqC
0kxzfIukFkSQVP3O2JBVQ2h7smcBtIYQfA3NJLjfWizF+I8dxSZJaz5wtSQWUtsCuAW7NZyCSpJwxZ0tSAaU
qsGOM38t3IJKk3DBnS1JhNVlghxBKSJZzOhpYRTJp5pcN7wwmSSoO5mxJKh7ZJjleDFwIPAM8T5K4r2uLoCR
JLWbOlqQika3A/i7wnRjjmBjjmcBRwHcyvSSSpOJizpakIpGtwN6apBekzpNAJ2BAPgOSJLWKOVuSikS2SY6
lJLfWBSDGWBNCWAl0ae3JQghDgRuBwcBs4NQY4/T1tLsYOBnYDPg3cEaM8dXWnleSNgLmbEkqEmlvNLPBQgh
lwFTgZqAncA1wbwihc6N2o0gude4H9AH+BvwlhNBmsUrSxs6cLUmt19wyfaNCCEsbtf9OCGFhw0YxxutTnGt
/oCbGOCnzekoI4RzgcOBPDdr1AcbFGN8CCCH8GrgC2Ap4J8V5JGljZc6WpCKQrcB+BxjTaNt8oPH6qrVAmmS
9C/Bao22R5NJjfbKOMf6iUZuRwCLgvRTnAGDlypXNN5Kk9sWcLUlFoskCO8a4XY7PtQmwvNG25UC3pt4QQvg
qcANwSoyxJu2JZsyY0aoAJenzypwtScUj7a3Sc2E50LXRtm7A0vW0JYRwPEkvy5kxxttbcqLBgwe3KkBJKrQ
iKjbN2ZLUjKZydlsW2DOBMxptC8A6iTiEcAlwNnBEjPHxlp6orKysVQFKkuqZsyWpldqywH4c6BJCOJPkEuL
xQH/gkYaNQgjfA84B9ooxzmrD+CRJa5izJamV2mwZpRjjKuBg4DjgY+BMYGSMcVkI4eEQwthM04uATYHnQwh
LGzwGtVWskrSxM2dLUuu1ZQ82McaXgb3Ws/3gBs93bsuYJEnrZ86WpNbxRgCSJElSDllgS5IkSTlkgS1JkiT
lkAW2JEmSlEMW2JIkSVIOWWBLkiRJOWSBLUmSJOWQBbYkSZKUQxbYkiRJUg5ZYEuSJEk5ZIEtSZIk5ZAFtiR
JkpRDFtiSJElSDllgS5IkSTlkgS1JkiTlkAW2JElqt2pqCh2BNkalhQ5AkiQpl6qrYf58WLgQqqqgtBT69IE
BA6CkpNDRaWNggS1JktqN6mqIEVasWLOtqiopuBcvhhAsspV/DhGRJEntxvz5axfXDa1YkeyX8s0CW5IktRs
LF27YfikXLLAlSVK7UFOTDAfJpqrKiY/KPwtsSZLULnTsmExozKa0NGkn5ZM/YpIkqd3o02fD9ku5YIEtSZL
ajQEDoGvX9e/r2jXZL+WbBbYkSWo3SkqSpfgGDFgzXKS0NHntEn1qK66DLUmS2pWSEhg4MHnUVNXQsdT+RLU
tC2xJktS+NLiVY8fVq6FzZ2/lqDZlgS1JktqP6mp47TV47z349NPkdUkJ9OwJH38Mu+5qka28s8CWJEntx/v
vw6xZsGrVmm3V1bBoESxdCj16wDbbFC4+bRQclCRJktqPxsV1Q6tWJfulPLPAliRJ7UNNTdJTnc2iRd7KUXl
ngS1JkiTlkAW2JElqHzp2THcrR++VrjzzJ0ySJLUfIUCXLuvf16VLsl/KMwtsSZLUfgwcCLvsAr17r1mOr6Q
keb3LLsl+Kc9cpk+SJLUfJSXJWtebbw4LF4I3mlEBWGBLkqT2Za17pdc45lptzp84SZLUfllcqwD8qZMkSZJ
yyAJbkiRJyiELbEmSJCmHLLAlSZKkHLLAliRJknLIAluSJEnKIQtsqSVqagodgSRJKnLeaEZqTnU1zJ+f3BF
s5UooK/OOYJIkqUkW2FI21dXwyivJ4/33obISOnVK7g62++7JwyJbkiQ1YIEtZfPOO/DEE7B06ZptlZUwd27
So92jB2y/fcHCkyRJxccx2FI206evXVzPn7/m+dKlyX5JUvF6441CR6CNUJv2YIcQhgI3AoOB2cCpMcZ1KpQ
QwtnA+cCmwAPAKTHGZW0Zq0RNDbz9Nrz+Otx1F6xevWZf587wjW8kw0VqaqCjf6uq/TFn63Pr2WfhmGNg3jy
orYUOHWDLLeHuu2H48EJHp41Am1UFIYQyYCpwM9ATuAa4N4TQuVG7w0gS9f7A1sDmwOVtFadUr6YGZs6EP/x
h7eIaktd/+EOy35VF1A6Zs/9/e/fP2lQUxnH8l/7BWoeMQrMWjnChS6BrcAjUQTt16OBSV/sGBF+Ar8AuOqk
lhCJFxLoIXQt3CaRyJJNCNsWAloai1+FYLLf513Lvud7b72dpe84DfYbw8CO55wS5dXAg1Wru3EwUubUocn/
Xam4fSJnPt91uS/ptrX1qrT2x1j6X9FXS3VjdfUnPrLWfrLU9SY8lPTDGcJIMfs3MSI3G6JpGw9UBxcPMRj6
trbmzMoOcnLh9IGU+k8EtSYexNSv30eNOrO51rKYsqSLp8yT/6Pj4+PJdAmdM9/ujC/p9/eL1hmJiZiOXprv
d0QXdLnMbqfMZsG9IOoqtHUmaH1N3+nu8bqh2u33h5oBzWi0tTVDW3t6WliapBHKFmY386XS0NO6xvShSe3d
XWlz00xOuJJ8B+0jS9djavKQfY+pOh3S8bqggCC7cHHBOEEgbG+PL1tc9NIOr4j8Km8xs5E8QuEPnp89eD1I
qKVhd9dcTCm3YzPYZsD9KehhbM5JeDagzsZqepDGf+fwzNzd3mf6A88plqdcbuT/L6w3FxMxGPi0suAONI/a
Z20ibz4D9QdI1Y8ympC25gzE3Jb2P1b2QtGWM2ZH0Re40+ktrLVc1wL9mU1pZGXxTyNSU2weKiZmNfGo23W0
hgw46zs4yt+GFt1tErLV9SXckrUv6JmlT0j1r7U9jzDtjzKO/dW8kPZH0Vu6AzHe5K6AA/+p1aW/PvZN9Vrn
s1uv1bPoCUsbMRm4tL0v7+1Kl4u6/ltzPSsWtcw82PChFo55TyqEwDKNqtZp1GyiqVosDjUhVGIaqVqulrPv
whZmN1HU6HGhEaobNbL5+DrgIwjUA5AvhGhkgYAMAAAAJImADAAAACSJgAwAAAAkiYAMAAAAJImADAAAACSJ
gAwAAAAkiYAMAAAAJImADAAAACZrJuoE0hGGYdQsAgAkxswEUTeG+Kh0AAADIEo+IAAAAAAkiYAMAAAAJImA
DAAAACSJgAwAAAAkiYAMAAAAJ+gP66CQ8Hqui0AAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x110d64470>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#Create Plot\n",
"fig = plt.figure(figsize=(12,6))\n",
"ax1 = fig.add_subplot(121)\n",
"ax2 = fig.add_subplot(122)\n",
"\n",
"#Plot Training\n",
"visualize_prob(model, x_train_cst, y_train, ax1)\n",
"ax1.set_title('Training Dataset')\n",
"\n",
"#Plot Testing\n",
"visualize_prob(model, x_test_cst, y_test, ax2)\n",
"ax2.set_title('Testing Dataset')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Does Logistic Regression perform better or worse? Has it overfitted to the
training set? What would points near 0.5 line mean?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Analyzing Significance of Coefficients\n",
"\n",
"How many of the coefficients estimated by the multiple logistic regression
model are significantly different from zero at a *significance level of 95%*? "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[https://www.sagepub.com/sites/default/files/upm-binaries/21122_Chapter_21.pdf]"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Columns : ['petal_width']\n",
"Count of 95% statistically significant coefficients : 1\n"
]
}
],
"source": [
"#Creating model\n",
"model = LogisticRegression(C=100000)\n",
"\n",
"#Initializing variables\n",
"bootstrap_iterations = 1000\n",
"coeffs = np.zeros((bootstrap_iterations, data_train.shape[1]-1))\n",
"\n",
"#Conduct bootstraping iterations\n",
"for i in range(bootstrap_iterations):\n",
" temp = data_train.sample(frac=1, replace=True)\n",
" response_variable = temp['species']\n",
" temp = temp.drop(['species'], axis=1)\n",
" model.fit(temp, response_variable) \n",
" coeffs[i,:] = model.coef_ \n",
"\n",
"#Find Significant Columns, Count\n",
"coeffs_count, significant_cols = 0, []\n",
"for i in range(coeffs.shape[1]):\n",
" coeff_samples = coeffs[:,i]\n",
" lower_bound = np.percentile(coeff_samples, 2.5)\n",
" upper_bound = np.percentile(coeff_samples, 97.5) \n",
" if lower_bound>0 or upper_bound<0:\n",
" coeffs_count += 1\n",
" significant_cols.append(data_train.columns[i])\n",
" \n",
"print('Columns :', significant_cols)\n",
"print('Count of 95% statistically significant coefficients :', coeffs_count)"
]
},
{
"cell_type": "markdown",
"metadata": {},
 "source": [
"# Part Four"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## - Use PCA to Reduce Dimensionality \n",
"## - Perform Classification on PCA Components"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Although PCA is used for dimensionality reduction, let's apply it to this
problem. Find the number of principal components needed to explain 85% of the data
variance and compare the performance of the model fitted to this data."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of Principal Components that explain >=85% of Variance: 2\n",
"Total Variance Explained by 2 components: 88.42744186872083%\n"
]
}
],
"source": [
"#Create and fit PCA object\n",
"pca = PCA()\n",
"pca.fit(x_train)\n",
"\n",
"#Transforming x_train and x_test \n",
"x_train_pca = pca.transform(x_train)\n",
"x_test_pca = pca.transform(x_test)\n",
"\n",
"#Find number of components that explain predefined variance threshold\n",
"sum_variance, component_count = 0, 0\n",
"while sum_variance < 0.85:\n",
" sum_variance += pca.explained_variance_ratio_[component_count]\n",
" component_count += 1\n",
" \n",
 "print('Number of Principal Components that explain >=85% of Variance: ',
component_count)\n",
"print('Total Variance Explained by '+str(component_count)+' components:',
str(sum_variance*100)+'%')"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training Set Accuracy: 100.0%\n",
"Testing Set Accuracy: 86.11111111111111%\n"
]
}
],
"source": [
"#Create and fit PCA object\n",
"pca = PCA(n_components=component_count)\n",
"pca.fit(x_train)\n",
"\n",
"#Transforming x_train and x_test \n",
"x_train_pca = pca.transform(x_train)\n",
"x_test_pca = pca.transform(x_test)\n",
"\n",
"#Add constant to x_train and x_test\n",
"x_train_pca_cst = sm.add_constant(x_train_pca)\n",
"x_test_pca_cst = sm.add_constant(x_test_pca)\n",
"\n",
"#Training\n",
"model = LogisticRegression(C=100000).fit(x_train_pca_cst, y_train)\n",
"\n",
"#Predict\n",
"y_pred_train = model.predict(x_train_pca_cst)\n",
"y_pred_test = model.predict(x_test_pca_cst)\n",
"\n",
"#Perfromance Evaluation\n",
"train_score = accuracy_score(y_train, y_pred_train)*100\n",
"test_score = accuracy_score(y_test, y_pred_test)*100\n",
"\n",
"print(\"Training Set Accuracy:\",str(train_score)+'%')\n",
"print(\"Testing Set Accuracy:\",str(test_score)+'%')"
]
},
{
"cell_type": "markdown",
 "metadata": {},
"source": [
"We observe that 2 PC's were enough to replicate the accuracy performance of
Logistic Regression with all predictors. Imagine the impact to complexity we can
cause when we have multiple thousand predictors but few PCA components are able to
able to match their performance. Drawback? Interpretability."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"image/png":
"iVBORw0KGgoAAAANSUhEUgAAAtgAAAFlCAYAAADRb58zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgA
ACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGx
vdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XmcVmXd+PEPzAgDooCshmsul4qSNKjhk5lL/R5R0Ud/alYWKYm
45ZKallkq1S/KyA2Uwqxwy9xwSS2z6HFJx9wQLlFxRURQQRiW2X5/nHuGYWTuOTPcG8Pn/XrNi/ucc93nfO8
7+853rnNd1+nS0NCAJEmSpNzoWuwAJEmSpM7EAluSJEnKIQtsSZIkKYcssCVJkqQcssCWJEmScsgCW5IkScq
h8mIHoM4jhPA74JttNDsgxvhoB8//I+CUGOPglO2/CPwd2DXGOKcj10x5nUeB/ZvtqgOWAk8Cl8YYH2/n+bY
GvhBjnJ6zINNddxTwTozxuUJeV1Lx5Tt/t7jWpsCJMcarMttjganAJjHG2vU9f5brvg0MabarFvgQeBS4OMY
Y23m+3YHtY4wzchZk29fsApwAPBRjXFCo66r97MFWLn0H2LLZz/vAL1vse2w9zv8LYI92tH8sc82563HNtO5
mzWf8NHAoSZH9aAjh4Hae6/fA4bkNL7sQwrbAfSTxS9r45Dt/N/c94IJm29OBLfNZXDcziTWfZyfgWGAg8Hg
IYWg7z/UAsE9uw2vTgcCNQK8CX1ftZA+2cibGuARY0rgdQqgHluXqr+wY4zJgWTvarwYK9Rf+yhaf880QwhP
AX4DrQgihHb88uuQ+vJK8pqQSke/83cJa+SbGuAJYkYfrrMvyFp/p9RDCIcDTwFUkBWxa5mq1ygJbBRdCGAN
cTtJrcTIwG/gvYCTwY2AvoCfwNjA5xjgx874f0WyISAihARgH/A/JEI33gT8D58cYa1sOEckM5agCNgWOARq
AfwLjY4zvZc65BfBrkh7orsCtJD0FNTHGMe35nDHGhhDCL4EHM59tZgihG/Ajkl6TbYCVJL1CZ8QY5zYfbhJ
COC7G2KWt92Tafjnzne5O8ovqb8DZMcZ3Msc3B36e+a56Ai+Q3BL9WwhhO2BeJuwHQgg3tvezStp4hBB2AyY
CXyDJNzOB78YY52WODwSuAb5Ikj9fBH4cY7w3hHA58P1MuwZgP2AXmg0RyQzluBIYARwCfEzSWXF6jLE6897
tSQri/YFq4PrM64dijJe35/PEGFeEEK4BrgkhfCrGOD+E0Af4GcnvgsEkdyT/monh/WbDTb4fQvhKjHHHtt6
TifsEkt77HYGPgBnAOTHGjzPHtyK5W/t/SH5HPUXyO+25zN3QhzNhzw0hXNzez6rCcYiIimUIsBtJAh1Pkow
eAiKwN0mh+Cfg5yGEbLfgfgHcBnyGJNmeDXw1S/vTgeXAvsCJJL0VP4WmsW0zgOHAESSJfyBwfEc+YMazmX8
/k/n3Z8AYks+8E0nBuxPJLxOAo4DHWTPkpM33ZP4ouJvkF9BQksS8MzCt2ed6gOQ7PQqobGyf6bl5i+Q7h+S
7+856fF5JnVimAJxJkjf2Ab4ErAb+HUJonB8zhSTHf4kkJ/0L+HMIYQhJPpsEvEuS455s5VKXAk8AnwV+QDL
u+KxMDD2BR0g6C/YnGVJ3EPD59fhojbl6WObf35P8njiOpBg+KfN5fpg5PjzzGSYBn0vznhDCZ0iGd0wCQqb
dISTfCSGEXsA/gG7NPs9s4InMeO9/knS0QNJp86v1+LzKM3uwVUyXxRhfhabeiMuAXzYOpQghXAycT5LwWkv
C02OMN2Re/yKEcBJJb/jvW2n/Wozx3MzrGEK4mTW3BPcjSY6fjTH+JxPDV4FXO/oBSSbQAPTO/Ps0cF+M8W+
Z7TdCCLeQFNDEGD8IIaxm7SEnWd9D0qtdQTIc5o0Y47wQwjFAv8zxAzOfa+sY49uZff8vhPBZkp6RB0II7zf
Gm7lVLEnrchqwMMZ4SuOOTK/sOyQF5QSSP/DnA/NijEtCCN8j6QD4OMa4LISwHKhvzHEhhHVd5+EY4xWZ13N
DCMeT5HaAr5B0yuwTY1yYOcf/Zc2duI5omasfBB5r/F1AknePJ1OAZ3qx60mGnCxK8x6SjpEuJJPJ38gcPxS
ozxz/GjAA+EpmiCPAWSGE/YDvxBi/HUJojHNRjHH5enxe5ZkFtoqpafJhpiicCpwSQtgD2IE1Saksyzlarg6
yhOSv/460HwGsaJYcG28d/jvL+drSmKw/ypzvphDC/iGEn5D0cASSnvzFrZ2grffEGJ8NIfyR5JbsZSGEvwH
3AzdnTvHZzL9zWvwi68aaXyqSlMZngZ1CCC3nw/QgyUsAl5D01L6fmYvyEPDHGOPSdlxnXbm6MZ+OAOY2Ftc
AmWEd6zOhfa1cDUwGDg0hfI3k99GuJH84ZJvo2dZ77iMZtnh/CGE+yXCPGcAdmeOfJRlS80GLXN2dZGigNiA
OEVExNU1qCSHsSjI85BjgdZLhD3umOMeqdezLNgkkW/taoEtmSEWuVGb+fQYgM87vHpJx4A8BY0nGMrYqzXt
ijCeQ9I78CNgEuBb43xBCBcn/z1eRfJ/Nf3Zjza1NSUqjK8lQhZb5JADnAsQY/wx8iqRHdhbwbeCFTE9sWm3
l6lzXL5UkY56fDSF0Be4lGepSD9xJ8rvpttbenOY9McYVMcYDSb6vX5N8R7dk2kLymebyye92V5LhJNqA2IO
tUjGeZCLLF2OMDQAhhMYe7ELNmv4PyVCLPTOvyUwwrCSZNNgRZ5IkzCdCCP2AU4FvxRh/19gghHAha3/Ghmb
H2nxP5ns6FTgzs67sVSGEg0gm1wwjmdDYHegdY6xqdo6fkfwSu6T5NSUpixdIhmi8k1n9g8wf8jcDfwwh3Ek
ypvj+GOOfgD+FEDYhGWp3DMn47fXNN/8BTg4hDGg2eXAASa9xu2Xy/HjgwRjjeyGEvUnGRu8XY/xXpk0Xksm
YzYdlNP8cI9p6TwjhAODQGON3gedI5hiNA6Zk5tK8AHydZCjNe83iu5Fkbs4UzNUbDAtslYo3SSa8jA4h/Id
kYkzj+LvuhQggxjgzhPBP4MYQwmnAB8CFwFa0ndQqmk3w2YRkXPQ5wMHAlzMrijQugzU6hPA4SW/FN0kmLTY
f9/wxsF1I1qZ+J8V7FpEk5W4hhJ9nYv0GyfCPOSTJ/RngphDCGcArJJMZzyeZONR4TYA9QghPxRhbHbIiaaN
2NcldtFszKzutIpk/cxBwXoyxPoSwC0nOGkdyR3I/kjHTjUMlPgZ6Z9q93oEYbiKZOHhLCOG8zL6JJB0kbeX
qTZvl6m4kPe8Xk/QmH5XZP5/kgWHHZYZybE4ywfIzJCtRNfqYZLjMp1K+ZxnJmOpqkiE0mwJHk+TpD0nmDp0
P3Jn5XAtJfo8cT7LKSuM1AYaHEN53zkzpcoiISsVVJMnlNySzpicC15HciizkQv7HZK5/H/C/JEXsE6z7dmV
zR5DMKH+XpICdTrJ01D4xxn8AZCZvHgVsTzJj/RGS5D6O5JdN46C7q4HtMnEMbOs9Mcb5wCiSISJPkiTzIcD
BMcalMcY64MskPUd/JLlleyxwQuPTImOMH5CMH7wMaJw0KklriTG+RrI83yYkK148RlJMHhhjfCXTbExm/y3
AyySrgJwdY7wlc/xWkk6V50hyV3tjWEWyWlItyQolf8nEMp+2c/VZrMnVc0jy3ovAiBjjy5nzv03S+fBl4CW
ScdJkPseuIYTNMtuNy+k9m7l21vfEGJ8iGepxWOaz/5Ok6B4VY2yIMX5E8sfIOyTDTf5Dckf1sMZe8cy17iH
J5T9K9YWpKLo0NHi3QYKm4RifJ7lNuLLZ/leAG2KME4oWnCQJgJCs3b9bjPH+ZvsqSHqBx8QYby1WbFIjh4h
Ia6wm6RX4QwhhUmbfWJLe4FYnt0iSCqoCuCeE8EOSsd+9SB7esoRkqTyp6BwiImVknqR1GMkDWZ4mGWqxF/C
lxicmSpKKK8Y4h2Si5dEkwztmkgxTOSAzzEIqOoeISJIkSTlkD7YkSZKUQxbYkiRJUg51ukmOVVVVjnmRtEG
rrKws1MOVis6cLWlDt66c3ekKbIDKysq2G0lSCaqqqmq7USdjzpa0oWotZztERJIkScohC2xJkiQphyywJUm
SpByywJYkSZJyyAJbkiRJyiELbEmSJCmHLLAlSZKkHLLAltrh/vuLHYEkqT0Wvrmy2CFoI9QpHzQj5dJ118E
pp3xy/5QpMG5c4eORJGU374Vl3PbVO+jz2jNsUreKmrLufPTpz3LsTUex/R69ih2eNgJFKbBDCHsDd8UYP9X
K8eOBCcBA4FHgpBjje4WLUEq0VlzDmv2duciur4eu3ufa6JmztSGZ98IyZuz3Uwas/KBp3yZ1qxgw93Fm7Bc
5fOaFnbrINm+XhoL+TxBC6BJCOBF4COjWSpthwBTgeGAAsAC4tmBBSs20VlynPb4hqquDd96B556D//wn+fe
dd5L9+TBv3jzGjx/PXnvtxfDhwxk9ejR/+tOf8nOxjClTpnDeeefl5dwPP/wwRx99dF7OXWjmbG2IbvvqHfR
qVlw312vlB9z21TsKHFH+FTJvm7PTKXQP9kXAsSQ9HRe00uZrwN0xxicBQggXAO+FEAbGGBemucjKlY63Uq6
Utdli5co8VZ5FUFcHL7/chRUr1uxbtQqWL4eFC2HnnRsoa/srSa2+vp6TTjqJI444gp/+9Kd069aNZ555hrP
PPpsePXpw8MEH5+5izYwZMwbIba6ora3lD3/4A9deey077rhjZ8lD5mxtcHq/+jRQn/X4ypXHFi6gPCtk3jZ
np1foAnsa8BNg/yxtdgEeb9yIMS4OIXyU2Z8qWc+aNWt9YpSAxgmNw9psN3HiLEaNyns4BbFw4SYsXrxJq8c
XLKhh4MCanF1vyZIlvPPOO+y00068+uqrAPTo0YPjjjuOefPmMWvWLG6//Xbmz5/P0qVLefXVV9lqq6046aS
T2G677QCYPXs206dPZ8GCBWy55ZZ885vfZMcddwTgrbfe4oYbbuD1119ns80245hjjuHzn/88t99+O2+//TZ
nnXUW9fX13HXXXTz66KOsXr2a4cOHc8IJJ9CzZ0+WL1/OlClTmDNnDj169GD33XdnzJgxdOv2yc7cqVOn8u6
773LIIYfw/PPPd5Y8ZM7WBmXhmyspr12RpbyG8voV/P2BKgZuU1GwuPKpkHnbnJ1eQQvsGOO7ACGEbM02Bap
b7KsGeqa9ztChQ9sdm9TS0KHwwx+23e688zrPf291dV3o3bv14+XlMHRoQ06vOWLECH75y19y6KGHMmLECHb
fffe1/j/8z3/+kzvvvJOf/exnHHTQQfz2t7/lqquuYsaMGSxatIgrrriCyy+/nC984Qs88sgjTJgwgXvuuYe
ePXty7rnncuSRR3LSSScRY2Ts2LF8+ctfZuDAgSxdupShQ4fyu9/9jhdeeIHp06ez2Wab8eMf/5i7776byy+
/nKuvvpo+ffrwj3/8gxUrVjB27FjmzZvHUUcd9YnPcdFFFzFw4EDuvvtu5s6d2+E8VErFpjlbG5qhQ+F35bf
SrX5Vq21Wd+3OAYdUFjCq/Cp03jZnr621nF2Kq4hUAz1a7OsJLEt7goqKzvFXqTYMFRWt9xxsSOrroayMNm8
lduuW2wk006ZN4+abb+ahhx5i2rRpAHz5y1/m4osvpm/fvpSXlzNy5EiOPPJIAM444wxuvvlmZs2axfPPP88
+++zDqMwthMaxgI8++ijbbrstK1as4Mwzz6Rr166MGDGCm2++ma233pry8nLKysqoqKjgrrvu4txzz23qXbn
gggs4+OCDmTBhAn379mXOnDn89a9/Zb/99uOuu+6iaysffptttgFgk002oUuXLhtTHjJnq6Qs2WEEA+Y+nvV
4Z/lvrhh525ydTikW2LOBpu6SEEJ/YIvMfqmgpkzJPpFxypTCxZJvXbsmPR21ta23KS/P/ez07t27M2bMGMa
MGcOqVauoqqpi4sSJXHTRRUyePBlYkwgBysrKGDBgAIsWLWL+/PnMnDmTESNGNB2vra2lsrKSXr16MXDgwLW
S66677vqJ67/77rucf/75lDX7DVVeXs78+fMZM2YMq1evZtq0aVx00UVUVlZy+eWXNyV2AeZslZhjbzqKGfv
FdU50XFaxBcfe9MnezA1VMfK2OTudUlzI5Wbg6BDC50MIFcBPgQdijIuLHJc2QuPGtV5Ed8Z1sPv3X7/j7XX
//fdz4IEH0tCQ3L7s3r07++67L+PHj2fOnDlN7RYuXDOUt7a2loULFzJ48GAGDBjAqFGjePrpp5t+ZsyYwdi
xYxk0aBALFy6kvn7NaMzp06fz3HPPrRXDgAEDuPbaa5ve//jjj3P33XezzTbbMHfuXI444ghmzJjBo48+Sr9
+/bjsssty+yVs+MzZKinb79GLw2deyPs7jaSmrDsANWXdeX+nkZ1yib5C5m1zdnolUWCHEKaEEKYAxBifBb5
NMrlmIfAp4FtFDE8buXHjoKEh+bnvvjWvO1txDTB4MPRoebM/o0eP5HgujRw5kurqaiZMmMDixYtpaGjgjTf
e4JZbbuGAAw5oajdz5kwee+wxampquOaaa+jbty/Dhw/n0EMP5e9//zuPP/44DQ0NVFVVMXr0aF544QWGDRt
G7969mTp1KrW1tTz//PNMmjSJXr3W/uV65JFHcs0117Bw4UJqamqYNGkSY8eOpaGhgdtuu41LLrmEZcuW0bd
vXyoqKujTp09uv4QNkDlbpW77PXpxwQvfYNzySRz28q8Yt3wSF7zwjU5XXENh87Y5O72iDBGJMT4K9G+2fUq
L47cBtxU4LKlNnWW1kNaUlUEIsGABLFqU3HYsL096QAYPbnucX3v17duXm266iUmTJnHYYYdRXV1Nv379OPz
wwznttNOa2g0bNoypU6dy+umnM3ToUK677jrKysrYbrvtmDRpEhMnTuT1119niy224MILL2TkyJEATJ48mUs
vvZSpU6fSr18/JkyYwA477LBWDOPGjaOmpobjjjuOpUuXsttuu3H99ddTXl7O2WefzcUXX8xBBx1ETU0Ne++
9N5dffnluv4QNgDlbG7LOslpIawqZt83Z6XVp7ObvLKqqqhoqKzvP7GCpmErhiWBXXXUVc+fO5corryxuIAV
SVVVFZWVll2LHUSjmbCm3ip23zdmJkhgiIqk0Fbu4liS1j3m7NPg/gyRJkpRDpbhMnyQ1OeOMM4odgiQpJXN
2wh5sSZIkKYcssCVJkqQcssCWJEmScsgCW5IkScohC2xJrWv2yFpJ0gbAvF0SXEVE0trq6gr3KEdJ0vozb5c
ce7AlrVFXBzEmibq2NtlXW5tsx5gcz7F58+Yxfvx49tprL4YPH87o0aP505/+lPPrNDdlyhTOO++8nJ7zscc
e47DDDmPPPffkq1/9KvPmzcvp+SVpnQqct83Z6VhgS1pjwQJYsWLdx1asSI7nUH19PWPHjmX33Xdn5syZVFV
V8YMf/ICJEyfy4IMP5vRazZ1yyilMnDgxZ+dbtGgRp59+Oueccw7//ve/2XfffTn33HNzdn5JalUB87Y5Oz0
LbElrLFq0fsfb6cMPP+Ttt99m9OjRVFRU0LVrV/bee2/OO+88ampqALjqqqs4++yz+cY3vsGee+7JMcccw+z
Zs5vO8dRTT3H00UczYsQIjjnmGJ5//vmmYy+//DJf//rXGT58OAcddBD33HNP0znPPPNMAOrq6rj66qs58MA
DGTlyJBdeeCHLli0DYOnSpZx66qnsvffeHHDAAXz/+99n1apVn/gcDz30ELvuuisHHngg3bp1Y/z48bz11lu
8+OKLOf2+JOkTCpi3zdnpWWBLStTXr7m92Jra2pxOoOnXrx9777033/rWt7jyyit54oknqK6u5phjjuGwww5
ravfAAw/wla98haeeeor999+fU089ldWrVzN//nzGjRvH+PHjeeKJJzjxxBP59re/zUcffcTq1asZN24cn/v
c53jyySf51a9+xSWXXMKrr766Vgw33HADDz/8MNOnT+fhhx9m5cqVXHbZZQBMmzaNsrIy/vWvf3HXXXcxa9a
spoTf3GuvvcYOO+zQtF1WVsbWW2/NK6+8krPvSpI+ocB525ydngW2pETXrsnEmGzKy5N2OfSb3/yGr3/96zz
xxBOMHTuWvffem3POOYcPP/ywqc3IkSMZNWoUm2yyCePHj6e6uppnnnmGe++9l3322YeDDz6Y8vJyDjnkEHb
eeWcefPBBnnnmGaqrqzn11FPp1q0bw4YN46abbmLQoEFrXf/222/n9NNPZ8stt6RXr15897vf5Z577mHVqlV
sttlmzJo1i/vuu4+amhruuOMOjjnmmE98hhUrVtCjR4+19vXo0YMVrd22laRcKELeNmen4yoiktbo3z/7eL3
+/XN+ye7duzNmzBjGjBnDqlWrqKqqYuLEiVx00UVMnjwZgG222aapfVlZGQMGDGDRokXMnz+fmTNnMmLEiKb
jtbW1VFZW0qtXLwYOHEjXZr9Ydt11109c/9133+X888+nrNlM+/LycubPn8+YMWNYvXo106ZN46KLLqKyspL
LL7+c7bbbbq1z9OjRg5UrV661b8WKFfTs2XO9vhtJalOB87Y5Ox17sCWtMXgwtPirvkmPHsnxHLr//vs58MA
DaWhoAJLEve+++zJ+/HjmzJnT1G7hwoVNr2tra1m4cCGDBw9mwIABjBo1iqeffrrpZ8aMGYwdO5ZBgwaxcOF
C6pvdGp0+fTrPPffcWjEMGDCAa6+9tun9jz/+OHfffTfbbLMNc+fO5YgjjmDGjBk8+uij9OvXr+lWZHOf/vS
n15qBXldXx5tvvsmOO+6Ys+9KktapgHnbnJ2eBbakNcrKIIQkITfediwvT7ZDyPl6qiNHjqS6upoJEyawePF
iGhoaeOONN7jllls44IADmtrNnDmTxx57jJqaGq655hr69u3L8OHDOfTQQ/n73//O448/TkNDA1VVVYwePZo
XXniBYcOG0bt3b6ZOnUptbS3PP/88kyZNolevXmvFcOSRR3LNNdewcOFCampqmDRpEmPHjqWhoYHbbruNSy6
5hGXLltG3b18qKiro06fPJz7Hl770JV588UUeeughVq9ezeTJkxk8eDC77bZbTr8vSfqEAuZtc3Z6DhGRtLa
yMhgyJPmpr8/5mOvm+vbty0033cSkSZM47LDDqK6upl+/fhx++OGcdtppTe2GDRvG1KlTOf300xk6dCjXXXc
dZWVlbLfddkyaNImJEyfy+uuvs8UWW3DhhRcycuRIACZPnsyll17K1KlT6devHxMmTFhrYgvAuHHjqKmp4bj
jjmPp0qXstttuXH/99ZSXl3P22Wdz8cUXc9BBB1FTU8Pee+/N5Zdf/onP0dij8pOf/IQLLriAXXfdlauuuoo
uXbrk7buTpCYFytvm7PS6NHbzdxZVVVUNlZWVxQ5DnVSe602tw1VXXcXcuXO58sorix1KQVRVVVFZWbnRVOb
mbKlzMWcn7MGW2uATaCVJUntYYEtZND6BtvnKPY1PoF2yJC/DkiVJ0gbOAlvKIs0TaIcMKWxMG5szzjij2CF
IklIyZyccTSplUeAnh0uSpE7AAltqRRGeHC5JkjoBC2ypFUV6crgkSdrAWRpIWbT1hNk8PDlckiRt4CywpSw
K/ORwSZLUCVhgS1kU+MnhkiSpE3CZPqkNBXxyuCRJ6gQsFaR2sLiWJEltsVyQJEmScsgCW5IkScohC2xJkiQ
phyywJUmSpByywJYkSZJyyAJbkiRJyiELbEmSJCmHLLAlSZKkHLLAliRJknLIAluSJEnKIQtsSZIkKYcssCV
JkqQcssCWJEmScsgCW5IkScohC2xJkiQphyywJUmSpByywJYkSZJyyAJbkiRJyqHyQl4shDAcuA4YCswFTok
xPrGOdj8AxgM9gX8D42OMrxUyVkna2JmzJaljCtaDHUKoAGYANwB9gCuBO0II3Vq0Oxz4JjACGAi8AvymUHF
KkszZkrQ+CjlE5ACgPsY4OcZYE2OcBiwGDm/RbudMXGVAF6AOWFHAOCVJ5mxJ6rBCDhHZBXipxb5Icuvxz83
23QKMA94iSdTzgf9qz4VWrlzZ8SglSWDOlqQOK2SBvSlQ3WJfNcmYvea6A/8CDgXeBX4F3BpC+K8YY0OaC82
aNWs9Q5WkjZ45W5I6qJAFdjXQo8W+nsCyFvuuBO6IMc4FCCGcCXwM7A68kOZCQ4cOXb9IJalISqjYNGdLUht
ay9kdKrBDCJuRTGiZFWNcmPJts4HTW54KuKnFvm1IekQa1QH1QE3a+CoqKtI2laROz5wtSYWVqsAOIewG/AH
4DvAi8BSwA7AihDA6xvi3FKd5BOgeQjgDmAKcAAwCHmzR7j7gvBDCX0jG8v00c82X08QqSRs7c7YkFVfaVUS
uAl4F5gAnAr2ALYEJwM/TnCDGuAo4BDge+AA4AxgdY1weQngghHBRpumPSCbQ/IskWe8AHBljrE8ZqyRt7Mz
ZklREXRoa2p6DEkJYDuweY5wXQvgHMDvGeEoIYVtgToyx5Ti9oqmqqmqorKwsdhiS1CFVVVVUVlZ2WZ9zmLM
lqTBay9lpe7A/BvqFEAYA+wL3Z/bvCryfmxAlSTlizpakIko7yfHPwO0ks8rfAf4SQvga8Gtgcp5ikyR1jDl
bkooobQ/2mcAk4CHg4BjjapKndv0w8yNJKh3mbEkqolQ92DHGOpJkDUAIoQswE3g77YMEJEmFYc6WpOJKu0z
fEJKHCUwgeXTuI8A+wMIQwqgY43/yF6IkqT3M2ZJUXGmHiFwDbAEsAr5JMlFmX5Jxfr/OT2iSpA4yZ0tSEaU
tsA8ETokxvgkcCcyIMT4JXEHydDBJUukwZ0tSEaUtsGuA8hBCL+CLrFny6VPAkjzEJUnqOHO2JBVR2mX6HgJ
uJFlbtRq4N4Tw38DVwJ15ik2S1DHmbEkqorQ92CeTPAZ3CXBojHEZyZi+O4Bz8hSbJKljzNmSVERpl+n7GDi
rxb5f5SUiSdJ6MWdLUnGlXaZvc5IHFww312B3AAAemUlEQVQleVgBQBegO/CZGOO2+QlPktRe5mxJKq60Q0S
mkiTrBuB/gNXANsBhwG/yE5okqYPM2ZJURGkL7C8DX4sxfhWYDfwixrgPcBWwe76CkyR1iDlbkooobYHdA5i
Tef0SsFfm9bXAF3IdlCRpvZizJamI0hbYc4GRmdezSR65C0kS3zTXQUmS1os5W5KKKO062L8Afh9C2AS4GXg
+hFAG7A38M1/BSZI6xJwtSUWUqgc7xngjcBAwO8b4MslEmU1JEvWJ+QtPktRe5mxJKq60PdjEGP+32eu/An/
NS0SSpPVmzpak4mm1wA4hPEWyxFObYox75ywiSVK7mbMlqXRk68G+t2BRSJLWlzlbkkpEqwV2jPHHhQxEktR
x5mxJKh1Zx2CHELYBfgj8OMb4VrP91wHdgO/HGOfnN0RJUhrmbEkqDa2uIhJC+DTwBPB5oHeLw/8mWVf135m
ELkkqInO2JJWObMv0XQY8AwyLMb7Y/ECM8bdAJfAycGn+wpMkpWTOlqQSka3APoDkNuPqdR2MMa4AfkSy1qo
kqbjM2ZJUIrIV2JsBi9t4/1tAn9yFI0nqIHO2JJWIbAX2bGBkG+8fCbyes2gkSR1lzpakEpGtwP4N8NMQwg7
rOhhC2BH4GfD7fAQmSWoXc7YklYhs62BfH0LYH3gxhHAn8BSwBOgL7AUcAdwPXFGIQCVJrTNnS1LpyLoOdoz
xayGE44BvAReQJOpFJIn7azHGO/IfoiQpDXO2JJWGrAU2QIzxVuDWAsQiSVpP5mxJKr5sY7AlSZIktZMFtiR
JkpRDFtiSJElSDllgS5IkSTnU5iTHRiGEPYC3Y4wfhhAOB/4v8DRwdYyxIV8BSpLaz5wtScWTqgc7hHAy8B9
gWAhhOHA7MBC4CLgsf+FJktrLnC1JxZV2iMh3gTExxn8AY4DnYoyHAF8hWW9VklQ6zNmSVERpC+ytgX9kXh8
GzMi8fhPoneugJEnrxZwtSUWUdgz2q8BhIYS3ge1Zk6zHAi/lIzBJUoeZsyWpiNIW2D8Ebs60nx5jfDaE8Gv
gROCIfAUnSeoQc7YkFVGqISIxxjuArYDKGOMJmd2TgU/HGB/JV3CSpPYzZ0tScaVepg9YCnwphPA/wK+BwcC
HeYlKkrS+zNmSVCRpl+nbHpgN/D+SZZ76AKcBszJLQEmSSoQ5W5KKK+0qIlcCDwPbAisz+44H7gN+lYe4JEk
dZ86WpCJKW2B/HrgixljfuCPGWAtcDlTmIzBJUoeZsyWpiNIW2KuALdax/9PAstyFI0nKAXO2JBVR2gL7D8D
VIYR9MtsDQgijgeuAm/ISmSSpo8zZklREaVcRuRD4CfAo0B14HKgFrs0cSyUzueY6YCgwFzglxvjEOtr9D/B
TYAgwCxgXY3wu7XUkaSNnzpakIkq7DnZtjPF8oC+wBzAc6BtjPCvGuDrNOUIIFSRPE7uBZEb7lcAdIYRuLdo
NB6YB3yZ5pO+dwJ/SfRxJkjlbkoqr1R7sEMIo4OEYY03mdUtDQggAxBjvT3GtA4D6GOPkzPa0EMLZwOHAn5u
1GwdMjTHOzMRxBfDXEELX5hN2JElrmLMlqXRkGyJyL8mDCRZmXremAShLca1dgJda7Isktx6bJ+vPAveGEB4
BhgH/AU5rT6JeuXJl240kqXMxZ0tSiWi1wI4xdl3X6/WwKVDdYl810LPFvi2A8SS9JC8APwbuCSHsnllmqk2
zZs1az1AlacNizpak0pH6UekhhOOBFTHGuzLbvwXujzH+Ofs7m1QDPVrs68knl4xaBdwRY3w6c52LgXNIelN
eTHOhoUOHpgxJkkpLropNc7Yk5V9rOTtVgR1COA+4GDi12e4FJGPyBsUYr01xmtnA6S1PzSeXjIokE2oadWn
2k0pFRUXappLU6ZizJam40vZgnwZ8pfnEmBjj90MIT5I8djdNsn4E6B5COAOYApwADAIebNHud8BNIYTfA0+
SPHlsLil7QiRJ5mxJKqa04/T6A6+uY/8cYMs0J4gxrgIOAY4HPgDOAEbHGJeHEB4IIVyUaXcPcAowNdNuH+C
IGGNDylglaWNnzpakIurS0NB2Dgwh/BV4Ezi5cdJKCKErcA0wNMb4hbxG2Q5VVVUNlZWVxQ5DkjqkqqqKysr
K1MMr1sWcLUmF0VrOTjtE5Bzgb8AbIYTnM/v2yLx/XeutSpKKx5wtSUWU9kmOz5PMCP8JyW3H2STj7HaKMT6
Tv/AkSe1lzpak4kq9TF+McXEI4U6SBw88CWwWY/w4b5FJkjrMnC1JxZN2mb5NgWnAMUA9sDNwRQihH3BUjPH
9/IUoSWoPc7YkFVfaVUR+DgwhueW4IrPvQqCCZMknSVLpMGdLUhGlLbCPAM6OMb7cuCPGOJtkaab/zkdgkqQ
OM2dLUhGlLbA3B5auY389sEnuwpEk5YA5W5KKKG2B/RBwcQihccx2QwhhAPAL4OG8RCZJ6ihztiQVUdoC+wx
gR2Ax0BNofIhBH+A7+QlNktRB5mxJKqJUq4jEGN8FPhdCOADYLfO+2cDDPg5XkkqLOVuSiiv1OtgZS4FlQB3
wrolakkqaOVuSiiDtOtifAu4CRgAfAGXA5iGEvwNHxxiX5C9ESVJ7mLMlqbjSjsGeSrKW6o4xxv4xxr7AriR
rql6dr+AkSR1izpakIkpbYO8PnBZjfK1xR2Z91dOB0fkITJLUYeZsSSqitAX2m8BO69g/AHgvd+FIknLAnC1
JRZR2kuMkYEoIYSfgX0AtMBy4GPhdCGFUY8MY4/05j1KS1B7mbEkqorQF9pTMvz9bx7GLmr1uIJlMI0kqHnO
2JBVR2nWw0w4lkSQVmTlbkoqrvetgE0LYBNgDWBhjfDv3IUmScsWcLUmFl7WXI4QwPoTwfAhh28z2MOBV4Gn
gjRDC9BBCtwLEKUlqgzlbkkpDqwV2COFE4ArgL0DjQwn+SNLrvRfJmqo7ARfmOUZJUhvM2ZJUOrINETkVODP
GOBUghDAC2B24IMZYldl3CXAV8ON8BypJysqcLUklIluBvSvwt2bbXyKZcX5vs32zga3yENd6efXVV1s91r9
/f3r37t20PW/ePOrr61ttv8UWW9C3b9+m7ddff526urpW2/fp04d+/fo1bb/11lusXr261fabb745AwYMaNp
+5513WLlyZavte/XqxaBBg5q23333Xaqrq1tt37NnT7bccsum7ffee49ly5a12r6iooIhQ4Y0bb///vssXbq
01fbdunVj6623btpevHgxH330Uavty8vL2XbbbZu2P/zwQz744INW23ft2pXtt9++aXvJkiUsWrSo1fYAO+y
wQ9Prjz/+mIULF2Ztv/3229O1a3IzZ/ny5SxYsCBr+2233Zby8uT/OitWrGD+/PlZ22+99dZ065bclV+1ahV
vv519GOyQIUOoqKgAoKamhjfffDNr+y233JKePXsCUFdXx+uvv561/aBBg+jVqxcADQ0NvPbaa1nbDxgwgM0
337xp+7XXXqOhoaHV9v369aNPnz5N2239f6Zv375sscUWTdtvvvkmNTU1rbbv3bs3/fv3b9p+++23WbVqVav
tN9tsMwYOHNi0PX/+fFasWNFq+0033ZTBgwc3bS9YsIDly5e32r5Hjx586lOfatpeuHAhH3/8cavtu3fvzlZ
brUmbixYtYsmSnDy53JyNOduc/UnmbHN2cwXL2Q0NDev82XnnnT/ceeedd222/bedd975zRZtPrfzzju/19o
5ivHz9NNPN5D8Ulnnz9VXX93Q3Oabb561/YQJE9Zqv9VWW2Vtf8EFF6zVfujQoVnbn3LKKWu133fffbO2P/7
449dqf8ghh2Rtf+ihh67V/rjjjsva/vOf//xa7U8++eSs7ffYY4+12p933nlZ22+zzTZrtb/sssuytu/Tp89
a7a+88sqs7bt06bJW+xtvvDFre6Bh6dKlTe3vuuuuNtu/9dZbTe0feeSRNtvPmjWrqX1VVVWb7R977LGm9q+
88kqb7R944IGm9u+//36b7W+99dam9qtWrWqz/fXXX7/Wd1pRUZG1/cSJE9dqP2jQoKztf/CDH6zVfqeddsr
a/swzz1yr/YgRI7K2/+Y3v7lW+4MOOihr+yOPPHKt9kcddVTW9gceeOBa7ceMGZO1fWVl5Vrtv/Od76x1/Om
nn25o6EDuM2cnP+Zsc3bLH3O2Obv5T6FydrZJjv8CTgQIIewCfAH4c4s2ZwJPZDmHJKkwzNmSVCK6NLRy2yC
EsCfwd2ARMBj4CBgRY3wvhHAIcBawH7Bf4/i+UlBVVdXQ/FZHS95u9HZjS95u9HZjc8W+3fjRRx9RWVnZpdU
TtMKcnTBnm7NbMmebs5srVM5utcAGCCFsCRwN1AO3xhgXZ/ZfAIwEJsQYn2r1BEVQVVXVUFlZWewwJKlDqqq
qOlRggzlbkgqttZyd9UEzMcZ3gavXsf//5TA2SVIOmLMlqTT4OF1JkiQphyywJUmSpByywJYkSZJyKNuj0v8
ZQhiUef2NEEL3woUlSWoPc7YklY5sPdiNj9kFuAHYPEtbSVJxmbMlqURkW0XkTuDhEEID0AVYEEJYZ8MYY1k
eYpMkpWfOlqQSka3A/jpwDdAXmAF8g+TBBZKk0mPOlqQS0WqBHWNsAB4DCCF8C7g9xtj6o3ckSUVjzpak0pH
1QTONYow3hhBGhhC+B+xKMnZ7DvDrGOPD+QxQktQ+5mxJKq5Uy/SFEP4vMBNYAVxF8qSwFcD9IYTR+QtPktR
e5mxJKq5UPdjAj4DvxRh/0WzfpBDCdzPH7slxXJKkjvsR5mxJKpq0D5r5NHDXOvbfRXL7UZJUOszZklREaQv
sV4H917H/AODt3IUjScoBc7YkFVHaISI/B64PIewO/BtoAEYC3wa+k6fYJEkdY86WpCJKu4rIHzIPLPgOMI5
ksswc4PgY4935C0+S1F7mbEkqrrQ92MQY/wD8IY+xSJJyxJwtScWTdgy2JEmSpBQssCVJkqQcssCWJEmScij
tkxz3yXcgkqTcMGdLUnGlneT4aAjhHWA6cFOMMeYxJknS+jFnS1IRpS2wBwFHA18BLgwhPEeSuG+JMS5Ie7E
QwnDgOmAoMBc4Jcb4RJb2JwI/jzH2T3sNSZI5W5KKKdUQkRjj0hjjDTHG/wMMAX4LfAmYG0J4KIRwQgihR7Z
zhBAqgBnADUAf4ErgjhBCt1bafxq4Iv1HkSSBOVuSiq0jkxw3B/oC/YFNgBrgbODNEMKhWd53AFAfY5wcY6y
JMU4DFgOHt2wYQigDfg9c34H4JElrmLMlqcBSDREJIWwDHEtyu3E48ARJMr0lxrg40+ZyYBrJrcl12QV4qcW
+SHLr8c8t9n8PmAXcD5yYJsbmVq5c2d63SFKnYc6WpOJKOwb7deBl4Cbg2Bjja+toM5MkkbdmU6C6xb5qoGf
zHSGESuAEYETmp91mzZrVkbdJUmfxOuZsSSqatAX254CnY4z1zXeGEMqB4THGp2KMDwIPZjlHNdByzF9PYFm
z8/UAbgTGxhiXhRBShre2oUOHduh9klRsOSo2zdmSVACt5ey0BfbjwGDg/Rb7dwD+QYsejVbMBk5vsS+Q9LA
0GgF8Grg3k6jLgZ4hhI+AYTHGN9MEW1FRkaaZJHVW5mxJKqJWC+wQwnjgx5nNLsBLIYSGFs16AS+mvNYjQPc
QwhnAFJJbioNo1oMSY5xJs8QfQvgicLtLPklSduZsSSod2XqwpwLLSVYamQZcBixpdryB5FbhI2kuFGNcFUI
4hCRR/wR4BRgdY1weQngAmBlj/En7P4IkCXO2JJWMLg0NLTs4PimEsD/wvzHG2vyHtH6qqqoaKisrix2GJHV
IVVUVlZWVXdbnHOZsSSqM1nJ2tiEiPwd+HGNcDhwKHNraBJYY4/m5ClSS1H7mbEkqHdmGiOxF8lCCxtetabs
LXJKUb+ZsSSoRrRbYMcYD1vVaklR6zNmSVDqyDREZlfIcDTHGB3IUjySpA8zZklQ6sg0RuTflORqAshzEIkn
qOHO2JJWIbENEuhYyEElSx5mzJal0ZBsishswJ8ZYn3ndmoYY4+zchyZJSsucLUmlI9sQkRdJHrW7MPO6geT
pYC15u1GSis+cLUklIluBvT3wfrPXkqTSZc6WpBKRbQz2Gy1fhxC2AHYBVgNzY4xLWnm7JKmAzNmSVDqy9WA
3CSFsClwPHMuaW4s1IYSpwHdjjCvzFJ8kqZ3M2ZJUXGlnnV8LDAO+BPQG+gKjM9u/zk9okqQOMmdLUhGl6sE
Gjga+GGN8utm+B0MIJwL3A+NyHpkkqaPM2ZJURGl7sN8HNm3l2Mc5ikWSlBvmbEkqorbWwW40GfhdCOFc4Em
gDvgMcCVwWV4jlCS1yZwtSaWjrXWwW66jentmH832Xwtcl/vQJEntYM6WpBLR1jrYkqQNgzlbkkpEqnWwG4U
QurNmyacuQHdgOPCJtpKkwjFnS1LpSLsO9heB37DuHpIaoCKHMUmS1oM5W5KKK+0qIlcALwGjgOXAUcAZwIf
AN/ITmiSpg8zZklREaQvs3YDvxRgfBJ4BVsQYrwVOB76br+AkSR1izpakIkpbYK8EVmVeR2DPzOvHgF1yHZQ
kab2YsyWpiNIW2P8CLgohbAY8DfxPCGET4AvAsnwFJ0nqEHO2JBVR2gL7XODzwMnAH4HNgKXAjcDV+QlNktR
B5mxJKqJUq4jEGCMQQgg9Y4wrQgh7A18EFscYn8xngJKk9jFnS1JxpSqwAUII5cCXQghDgdXAbODf+QpMktR
x5mxJKp6062DvAPwN2AKYSzK0ZCfgtRDCETHGefkLUZLUHuZsSSqutGOwbwCeAIbEGCtjjMOBrYHXgSl5ik2
S1DHmbEkqorQF9l7AJTHGjxt3xBg/BC4E9stHYJKkDjNnS1IRpS2wnwM+t479w0jWWJUklQ5ztiQVUatjsEM
IpzbbfAKYHEKoBJ4E6oDPkDwVbGJeI5QktcmcLUmlI9skx/NabL8HHJ75abQIGANcmtuwJEntZM6WpBLRaoE
dY9y+kIFIkjrOnC1JpaM962BvDpwE7Eoydns28IcY48I8xSZJ6iBztiQVT6pJjiGEYSRrqZ4N9AP6A2cBL4U
QdstfeJKk9jJnS1Jxpe3BngQ8CJwUY6wBCCFsAkwDrgD+Oz/hSZI6wJwtSUWUdpm+zwE/aUzUAJnXPwX+Kx+
BSZI6zJwtSUWUtsBeCGy1jv1bActzF44kKQfM2ZJURGmHiPwRmBpCOAv4N9AAjCS51XhTnmKTJHWMOVuSiih
tgX0psCVwO2t6vWuBa4Hv5SEuSVLHmbMlqYjSFth7AeOBc4AArAReiTFW5yswSVKHmbMlqYjSFth3AwfHGJ8
lud0oSSpd5mxJKqK0kxxfI+kFkSSVPnO2JBVR2h7s2cD0EML3gddJbjc2iTEem+O4JEkdZ86WpCJKW2DXA3/
IZyCSpJwxZ0tSEaUqsGOM38p3IJKk3DBnS1JxtVpghxDKSJZzOhpYRTJp5pfNnwwmSSoN5mxJKh3ZJjn+ALg
AeBJ4miRxX1OIoCRJ7WbOlqQSka3A/gbw9Rjj+BjjGcBRwNczvSSSpNJizpakEpGtwN6apBek0aPAJsDgfAY
kSeoQc7YklYhskxzLSR6tC0CMsT6EsBLo3tGLhRCGA9cBQ4G5wCkxxifW0e4HwMnA5sCzwOkxxhc7el1J2gi
YsyWpRKR90Mx6CyFUADOAG4A+wJXAHSGEbi3ajSG51flFoD/wV+C+EELBYpWkjZ05W5I6rq1l+saEEJa1aP/
1EMKi5o1ijNemuNYBQH2McXJme1oI4WzgcODPzdr1BybEGF8DCCH8GrgM2Ap4M8V1JGljZc6WpBKQrcB+Exj
fYt8CoOX6qg1AmmS9C/BSi32R5NZjU7KOMf6iRZvRwGLg7RTXAGDlypVtN5KkzsWcLUklotUCO8a4XY6vtSl
Q3WJfNdCztTeEEL4ATAHGxRjr015o1qxZHQpQkjZU5mxJKh1pH5WeC9VAjxb7egLL1tGWEMIJJL0sZ8QYb2r
PhYYOHdqhACWp2Eqo2DRnS1IbWsvZhSywZwOnt9gXgE8k4hDCxcBZwBExxkfae6GKiooOBShJamLOlqQOKmS
B/QjQPYRwBsktxBOAQcCDzRuFEL4FnA3sG2OcU8D4JElrmLMlqYMKtoxSjHEVcAhwPPABcAYwOsa4PITwQAj
hokzTC4HNgKdDCMua/exaqFglaWNnzpakjitkDzYxxueBfdex/5Bmr3cuZEySpHUzZ0tSx/ggAEmSJCmHLLA
lSZKkHLLAliRJknLIAluSJEnKIQtsSZIkKYcssCVJkqQcssCWJEmScsgCW5IkScohC2xJkiQphyywJUmSpBy
ywJYkSZJyyAJbkiRJyiELbEmSJCmHLLAlSZKkHLLAliRJknLIAluSJEnKIQtsSZIkKYcssCVJkqQcssCWJEm
ScsgCW5IkScohC2xJkiQphyywJUmSpByywJYkSZ1XfX2xI9BGqLzYAUiSJOVUXR0sWACLFkFtLZSXQ//+MHg
wlJUVOzptBCywJUlS51FXBzHCihVr9tXWJgX3kiUQgkW28s4hIpIkqfNYsGDt4rq5FSuS41KeWWBLkqTOY9G
i9Tsu5YAFtiRJ6hzq65PhINnU1jrxUXlngS1JkjqHrl2TCY3ZlJcn7aQ88r8wSZLUefTvv37HpRywwJYkSZ3
H4MHQo8e6j/XokRyX8swCW5IkdR5lZclSfIMHrxkuUl6ebLtEnwrEdbCl9qivd+yeJJW6sjIYMiT5MW+rCCy
wpbY0fyLY6tXQrZtPBJOkDYXFtYrAAlvKpq4OXnoJ3n4bPvoo2S4rgz594IMPYLfdLLIlSdJaLLClbN55B+b
MgVWr1uyrq4PFi2HZMujdG7bZpnjxSZKkkuN9EymblsV1c6tWJcclSZKascCWWlNfn/RUZ7N4sU8EkyRJa7H
AliRJknLIAltqTdeu6Z4I5gx1SZLUjJWBlE0I0L37uo91754clyRJasYCW8pmyBDYZRfo12/NcnxlZcn2Lrs
kxyVJkppxmT4pm7KyZK3rLbbwQTOSJCkVC2ypLT5yV5IktYOVgtQeFteSJKkNVguSJElSDllgS5IkSTlkgS1
Jkjovn7arInCSo9Qezz4Le+5Z7CgkSdnU1cGCBbBoEfUvv0LXnXd09ScVVEEL7BDCcOA6YCgwFzglxvjEOtq
dBZwHbAbcA4yLMS4vZKxSk7/8BY49Fj7+eM2+zTaD226D//7v4sUl5Zk5WxukujrqbrmV1WedQ/kHi6Chgbo
uXajdoj/dJl1B2VeOs8hW3hVsiEgIoQKYAdwA9AGuBO4IIXRr0e4wkkR9ALA1sAVwaaHilNbyl7/AqFFrF9e
QbI8alRyXOiFztjZUdffcR92YMZQvfh8aGpKdDQ2UL36fujFjqLvnvuIGqI1CIcdgHwDUxxgnxxhrYozTgMX
A4S3anQD8Nsb4coxxCXAxcFIIwT83VXjHHrsmQbfU0JAclzonc7Y2SKtPHkeXurp1HutSV8fqk8cVOCJtjAo
5RGQX4KUW+yLJrcc/t2h3Z4s2vYEhwJtpLrRy5cqORyk1U9ay57qljz+mzv/e1DmZs7Xhqa+n/IP3szYp/+B
9VlZX+1wD5VUhC+xNgeoW+6qBnm20a3zdsl2rZs2a1e7gpE949lmGpWg2a/p0Jz6qMzJna4NT//Ir7NnWqiE
NDbxw94xk4qOUJ4UssKuBHi329QSWtdGuMUm3bNeqoUOHtjs46ROGDoWxY9tu9rWvFSCY/9/eHatGEUVhAP4
nm8RYbB0S0k8RrBYCabSNjWLpO5gX8A18gzyAWoSkEiM2QtJPLVOLYiVaiIgrsRgR2c1KlNkdd/N95dwLc4r
h8HO59w5XxX8UNvVs5s/2drMyPWlrX5IURW7cuzu7mlhok3r2LAP26yQPRp6VSZ5eMK8cmfMpybvLvmhtbe1
f6oNx/f74AceR8RXfG4tJz2YufVnfzPL7txPHh+ubue6bY8pmGbBfJblWluV+koM0B2PWk7wcmfc4yUFZlsd
J3qQ5jf6krms3xTN7h4fNbSEXrYYURTMOi0nPZi6tHh/l+62bKYbfxsbOl1eyenzUQVVcNTPb4V/X9dckt5P
cT/IhyX6SO3Vdfy7L8kVZlg9/znuW5FGS52kOyHxMcwUUzN7eXnJy0qxk/67fb567B5sFpWczr3q7O+mdnmW
4sdVsFymKZGkpw42t9E7P0tvd6bpEroDi/E/7lOZQVVXng8Gg6zJYVP7kyJRVVZXBYFB0Xces6NlM268/OcI
UTOrZ7qiBvyFcA8wV4ZouCNgAANAiARsAAFokYAMAQIsEbAAAaJGADQAALRKwAQCgRQI2AAC0SMAGAIAWLXd
dwDRUVdV1CQBckp4NLJqF+1U6AAB0yRYRAABokYANAAAtErABAKBFAjYAALRIwAYAgBb9AFhuAxsAuDJnAAA
AAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x111412ac8>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#Create Plot\n",
"fig = plt.figure(figsize=(12,6))\n",
"ax1 = fig.add_subplot(121)\n",
"ax2 = fig.add_subplot(122)\n",
"\n",
"#Plot Training\n",
"visualize_prob(model, x_train_pca_cst, y_train, ax1)\n",
"ax1.set_title('Training Dataset')\n",
"\n",
"#Plot Testing\n",
"visualize_prob(model, x_test_pca_cst, y_test, ax2)\n",
"ax2.set_title('Testing Dataset')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>Predicted</th>\n",
" <th>False</th>\n",
" <th>True</th>\n",
" <th>All</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Actual</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>False</th>\n",
" <td>20</td>\n",
" <td>4</td>\n",
" <td>24</td>\n",
" </tr>\n",
" <tr>\n",
" <th>True</th>\n",
" <td>1</td>\n",
" <td>11</td>\n",
" <td>12</td>\n",
" </tr>\n",
" <tr>\n",
" <th>All</th>\n",
 " <td>21</td>\n",
" <td>15</td>\n",
" <td>36</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Predicted False True All\n",
"Actual \n",
"False 20 4 24\n",
"True 1 11 12\n",
"All 21 15 36"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Get Performance by Class\n",
"pd.crosstab(y_test, y_pred_test, margins=True, rownames=['Actual'],
colnames=['Predicted'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Where does PCA perform better/worse? Does trying to explain variance sacrifice
the power of the predictors? Is the variance we ignore, the last 15%, better for
generalizability of the model i.e. can it prevent overfitting to the training set?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[https://giphy.com/gifs/pca-Lyejb62QjQepG]"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
 },
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
}
},
"nbformat": 4,
"nbformat_minor": 1
}