Exploratory Data Analysis and Modeling for
Insurance and User Data
Submitted by- Ayushkumar Parmar [21IM30005]
Introduction
This Jupyter Notebook conducts exploratory data analysis (EDA) and modeling for two datasets:
insurance and user_data. The analysis includes visualizations, linear regression modeling for
insurance charges prediction, and logistic regression modeling for binary classification in user
data. Insurance Dataset
The exploration starts with loading and understanding the insurance dataset:
# Importing the libraries
import pandas as pd
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
user_data = pd.read_csv('/content/User_Data.csv')
insurance = pd.read_csv('/content/insurance.csv')
df = insurance.copy()
print('Rows and columns in the dataset:', df.shape)
df.head() # Top 5 rows
df.info()
Rows and columns in the dataset: (1338, 7)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1338 entries, 0 to 1337
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 age 1338 non-null int64
1 sex 1338 non-null object
2 bmi 1338 non-null float64
3 children 1338 non-null int64
4 smoker 1338 non-null object
5 region 1338 non-null object
6 charges 1338 non-null float64
dtypes: float64(2), int64(2), object(3)
memory usage: 73.3+ KB
Fitting a Line - Charges vs. BMI
A linear regression line is fitted to understand the relationship between BMI and insurance
charges:
sns.lmplot(x='bmi', y='charges', data=df, aspect=2, height=5)
plt.xlabel('Body Mass Index: as independent variable')
plt.ylabel('Insurance Charges: as dependent variable')
plt.title('Charges vs. BMI')
Text(0.5, 1.0, 'Charges vs. BMI')
Exploratory Data Analysis (EDA)
EDA includes descriptive statistics, checking for missing values, and exploring correlations:
df.describe()
# Check for missing values
missing_values = df.isnull()
# Summarize the missing values
missing_values_count = missing_values.sum()
# Display the result
print("Missing values in each column:")
print(missing_values_count)
# Correlation heatmap
corr = df.corr()
sns.heatmap(corr, cmap='viridis', annot=True)
Missing values in each column:
age 0
sex 0
bmi 0
children 0
smoker 0
region 0
charges 0
dtype: int64
<Axes: >
Distribution and Relationships Visualization
Visualizations include distribution plots, violin plots, box plots, and scatter plots:
# Distribution of insurance charges
f = plt.figure(figsize=(12,4))
sns.distplot(df['charges'], bins=50, color='r', ax=f.add_subplot(121))
sns.distplot(np.log10(df['charges']), bins=40, color='b',
ax=f.add_subplot(122))
f.suptitle('Distribution of insurance charges')
# Violin plots
f = plt.figure(figsize=(13,6))
sns.violinplot(x='sex', y='charges', data=df, palette='Wistia',
ax=f.add_subplot(121))
sns.violinplot(x='smoker', y='charges', data=df, palette='magma',
ax=f.add_subplot(122))
# Box plot
plt.figure(figsize=(13,6))
sns.boxplot(x='children', y='charges', hue='sex', data=df,
palette='rainbow')
# Scatter plots
f = plt.figure(figsize=(13,6))
sns.scatterplot(x='age', y='charges', data=df, palette='magma',
hue='smoker', ax=f.add_subplot(121))
sns.scatterplot(x='bmi', y='charges', data=df, palette='viridis',
hue='smoker', ax=f.add_subplot(122))
<Axes: xlabel='bmi', ylabel='charges'>
Dummy Variable and Log Transform
Creating dummy variables for categorical columns and log-transforming the charges:
# Dummy variable
categorical_columns = ['sex', 'smoker', 'region']
df_encode = pd.get_dummies(data=df, prefix='OHE', prefix_sep='_',
columns=categorical_columns,
drop_first=True,
dtype='int8')
# Log transform
df_encode['charges'] = np.log(df_encode['charges'])
Linear Regression Model
Building and evaluating a linear regression model:
# Train test split
from sklearn.model_selection import train_test_split
X = df_encode.drop('charges', axis=1) # Independent variable
y = df_encode['charges'] # Dependent variable
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.3, random_state=23)
# Model building
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(X_train, y_train)
# Get the coefficients and the intercepts
coefficients = lin_reg.coef_
intercept = lin_reg.intercept_
# Print them
coefficients_df = pd.DataFrame({'Feature': X_train.columns,
'Coefficient': coefficients})
print("Intercept: ", intercept)
print("Coefficients:")
print(coefficients_df)
# sklearn regression module
y_pred_sk = lin_reg.predict(X_test)
# Evaluation: MSE and R-squared
from sklearn.metrics import mean_squared_error
J_mse_sk = mean_squared_error(y_pred_sk, y_test)
R_square_sk = lin_reg.score(X_test, y_test)
# Adjusted R-squared
n = len(y_test)
p = X_test.shape[1]
adjust_r_squared = 1 - (1 - R_square_sk) * ((n-1)/(n-p-1))
print("Adjusted Rsquared: ", adjust_r_squared)
Intercept: 7.07542806626153
Coefficients:
Feature Coefficient
0 age 0.033057
1 bmi 0.013706
2 children 0.101695
3 OHE_male -0.069942
4 OHE_yes 1.547184
5 OHE_northwest -0.054662
6 OHE_southeast -0.145206
7 OHE_southwest -0.135254
Adjusted Rsquared: 0.7722401938297871
Model Validation and Checks
Checking for linearity, residual normality, mean, multivariate normality, homoscedasticity, and
multicollinearity:
# Check for linearity and residual normality
f = plt.figure(figsize=(13,5))
ax = f.add_subplot(121)
ax.scatter(y_test, y_pred_sk, color='r')
ax.set_title('Check for linearity: Actual vs. Predicted')
ax = f.add_subplot(122)
sns.distplot((y_test - y_pred_sk), ax=ax, color='b')
ax.axvline((y_test - y_pred_sk).mean(), color='k', linestyle='--')
ax.set_title('Check for residual normality and mean: Residual error')
ax.set_xlabel('Residuals')
# Check for Multivariate Normality
f, ax = plt.subplots(1,2,figsize=(13,6))
_,(_,_,r) = sp.stats.probplot((y_test - y_pred_sk), fit=True,
plot=ax[0])
ax[0].set_title('Check for Multivariate Normality: Q-Q plot')
# Check for Homoscedasticity
sns.scatterplot(y=(y_test - y_pred_sk), x=y_pred_sk, ax=ax[1],
color='r')
ax[1].set_title('Check for Homoscedasticity: Residual vs. Predicted')
# Check for Multicollinearity - Variance Inflation Factor
VIF = 1 / (1-R_square_sk)
print('VIF: ', VIF)
VIF: 4.479966203229661
Logistic Regression for User Data
Moving on to the user data, logistic regression is performed for binary classification:
df = user_data.copy()
# Train test split
X = df.iloc[:,2:3].values
y = df.iloc[:,4].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size =
0.25, random_state = 0)
# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)
# Fitting Logistic Regression on the training set
from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state=0)
classifier.fit(X_train, y_train)
# Predicting the test set results
y_pred = classifier.predict(X_test)
# Model Evaluation - Classification Report, Accuracy, and Confusion
Matrix
from sklearn.metrics import classification_report, accuracy_score,
confusion_matrix
report = classification_report(y_test, y_pred)
accuracy = accuracy_score(y_test, y_pred)
cm = confusion_matrix(y_test, y_pred)
# Display results
print("Classification report:\n", report)
print("Accuracy: ", accuracy)
# Confusion Matrix Visualization
cm_df = pd.DataFrame(cm, index=['Actual 0', 'Actual 1'],
columns=['Predicted 0', 'Predicted 1'])
sns.heatmap(cm_df, annot=True, fmt='d', cmap='Blues')
Classification report:
precision recall f1-score support
0 0.89 0.97 0.93 68
1 0.92 0.75 0.83 32
accuracy 0.90 100
macro avg 0.91 0.86 0.88 100
weighted avg 0.90 0.90 0.90 100
Accuracy: 0.9
<Axes: >
Conclusion
This comprehensive analysis provides insights into the relationships within the insurance
dataset and performs a binary classification on the user data using logistic regression.
Visualizations, model building, and evaluations contribute to a thorough understanding of the
datasets.
