10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

Workflow stages
The competition solution workflow goes through seven stages described in the Data Science Solutions book.

1. Question or problem definition.

2. Acquire training and testing data.
3. Wrangle, prepare, cleanse the data.
4. Analyze, identify patterns, and explore the data.
5. Model, predict and solve the problem.
6. Visualize, report, and present the problem solving steps and final solution.
7. Supply or submit the results.

The workflow indicates general sequence of how each stage may follow the other. However there are use
cases with exceptions.

We may combine mulitple workflow stages. We may analyze by visualizing data.

Perform a stage earlier than indicated. We may analyze data before and after wrangling. Perform a
stage multiple times in our workflow. Visualize stage may be used multiple times. Drop a stage
altogether. We may not need supply stage to productize or service enable our dataset for a competition.

Question and problem definition

Competition sites like Kaggle define the problem to solve or questions to ask while providing the datasets for
training your data science model and testing the model results against a test dataset. The question or problem
definition for Titanic Survival competition is described here at Kaggle (https://www.kaggle.com/c/titanic).

Knowing from a training set of samples listing passengers who survived or did not survive the
Titanic disaster, can our model determine based on a given test dataset not containing the
survival information, if these passengers in the test dataset survived or not.

We may also want to develop some early understanding about the domain of our problem. This is described on
the Kaggle competition description page here (https://www.kaggle.com/c/titanic). Here are the highlights to note.

On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502
out of 2224 passengers and crew. Translated 32% survival rate.
One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for
the passengers and crew.
Although there was some element of luck involved in surviving the sinking, some groups of people were
more likely to survive than others, such as women, children, and the upper-class.

Workflow goals
The data science solutions workflow solves for seven major goals.

Classifying. We may want to classify or categorize our samples. We may also want to understand the
implications or correlation of different classes with our solution goal.

Correlating. One can approach the problem based on available features within the training dataset. Which
features within the dataset contribute significantly to our solution goal? Statistically speaking is there a
correlation (https://en.wikiversity.org/wiki/Correlation) among a feature and solution goal? As the feature
values
localhost:8888/notebooks/titanicMain (3).ipynb 1/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook
 change does the solution state change as well, and visa-versa? This can be tested both for numerical and
categorical features in the given dataset. We may also want to determine correlation among features other than
survival for subsequent goals and workflow stages. Correlating certain features may help in creating,
completing, or correcting features.

Converting. For modeling stage, one needs to prepare the data. Depending on the choice of model algorithm
one may require all features to be converted to numerical equivalent values. So for instance converting text
categorical values to numeric values.

Completing. Data preparation may also require us to estimate any missing values within a feature. Model
algorithms may work best when there are no missing values.

Correcting. We may also analyze the given training dataset for errors or possibly innacurate values within
features and try to corrent these values or exclude the samples containing the errors. One way to do this is to
detect any outliers among our samples or features. We may also completely discard a feature if it is not
contribting to the analysis or may significantly skew the results.

Creating. Can we create new features based on an existing feature or a set of features, such that the new
feature follows the correlation, conversion, completeness goals.

Charting. How to select the right visualization plots and charts depending on nature of the data and the solution
goals.

2
Type Markdown and LaTeX: ��

In [1]: 

# data analysis and wrangling

import pandas as pd
import numpy as np
import random as rnd

# visualization
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

# machine learning
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import Perceptron
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier

Acquire data
The Python Pandas packages helps us work with our datasets. We start by acquiring the training and testing
datasets into Pandas DataFrames. We also combine these datasets to run certain operations on both datasets
together.

localhost:8888/notebooks/titanicMain (3).ipynb 2/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook
 In [ ]: 

In [2]: 

train_df = pd.read_csv('train.csv')
test_df = pd.read_csv('test.csv')
combine = [train_df, test_df]
newdf = pd.concat(combine)
newdf.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1309 entries, 0 to 417
Data columns (total 12 columns):
Age 1046 non-null float64
Cabin 295 non-null object
Embarked 1307 non-null object
Fare 1308 non-null float64
Name 1309 non-null object
Parch 1309 non-null int64
PassengerId 1309 non-null int64
Pclass 1309 non-null int64
Sex 1309 non-null object
SibSp 1309 non-null int64
Survived 891 non-null float64
Ticket 1309 non-null object
dtypes: float64(3), int64(4), object(5)
memory usage: 132.9+ KB

Analyze by describing data

Pandas also helps describe the datasets answering following questions early in our project.

Which features are available in the dataset?

Noting the feature names for directly manipulating or analyzing these. These feature names are described on
the Kaggle data page here (https://www.kaggle.com/c/titanic/data).

In [3]:  print(train_df.columns.values)

['PassengerId' 'Survived' 'Pclass' 'Name' 'Sex' 'Age' 'SibSp'

'Parch' 'Ticket' 'Fare' 'Cabin' 'Embarked']

In [4]: 

# using newdf to print

print(newdf.columns.values)

['Age' 'Cabin' 'Embarked' 'Fare' 'Name' 'Parch' 'PassengerId'

'Pclass' 'Sex' 'SibSp' 'Survived' 'Ticket']

localhost:8888/notebooks/titanicMain (3).ipynb 3/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

Which features are categorical?

 These values classify the samples into sets of similar samples. Within categorical features are the values
nominal, ordinal, ratio, or interval based? Among other things this helps us select the appropriate plots for
visualization.

Categorical: Survived, Sex, and Embarked. Ordinal: Pclass.

Which features are numerical?

Which features are numerical? These values change from sample to sample. Within numerical features are the
values discrete, continuous, or timeseries based? Among other things this helps us select the appropriate plots
for visualization.

Continous: Age, Fare. Discrete: SibSp, Parch.

In [4]: 

# preview the data

train_df.head()

Out[4]:

PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare C Braund,

01031211231334114503 Futrelle,
Mrs. female 26.0 0 0STON/O2. 3101282 7.9250
Jacques Heath
(Lily May Peel)

Mr. Owen Harris Allen, Mr. William

Henry female 35.0 1 0 113803 53.1000 male 35.0 0
Cumings, Mrs. John Bradley male 22.0 1 0 A/5 21171 7.2500
(Florence Briggs
Th...

Heikkinen, Miss. 0 373450 8.0500

Laina female 38.0 1 0 PC 17599 71.2833

localhost:8888/notebooks/titanicMain (3).ipynb 4/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [6]: 

# preview the data using newdf

newdf.head()

Out[6]:

Age Cabin Embarked Fare Name Parch PassengerId Pclass Sex SibSp Sur Braund,

0 22.0 NaN S 7.2500 1 38.0 C85 C 71.2833 2 26.0 NaN S 7.9250 3 35.0 C123 S 53.1000 4 35.0 NaN S 8.0500
Mr. Owen Harris Laina Henry
0 1 3 male 1 0 2 1 female 1 0 3 3 female 0 0
Cumings, Mrs. John Bradley Futrelle,
(Florence Briggs Mrs.
Th... Jacques Heath
(Lily May Peel)
Heikkinen, Miss.
4 1 female 1 0 5 3 male 0
Allen, Mr. William

Which features are mixed data types?

Numerical, alphanumeric data within same feature. These are candidates for correcting goal.

Ticket is a mix of numeric and alphanumeric data types. Cabin is alphanumeric.

Which features may contain errors or typos?

This is harder to review for a large dataset, however reviewing a few samples from a smaller dataset may just
tell us outright, which features may require correcting.

Name feature may contain errors or typos as there are several ways used to describe a name including
titles, round brackets, and quotes used for alternative or short names.

localhost:8888/notebooks/titanicMain (3).ipynb 5/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [5]:  train_df.tail()

Out[5]:

PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cab Montvila,

886 887 0 2 887 888 1 1 Margaret Edith

female NaN 1 2 W./C.
Johnston, Miss.
Catherine Helen
"Carrie"

888 889 0 3 Behr, Mr. Karl

Howell

Dooley,
Mr.
889 890 1 1 890 891 0 3 Patrick
male 27.0 0 0 211536 13.00 Na female 19.0
6607 23.45 Na

Rev.
Juozas 0 0 112053 30.00 B
male 26.0 0 0 111369 30.00 C1 male 32.0 0
Graham, Miss.
0 370376 7.75 Na

In [8]:  newdf.tail()

Out[8]:

Age Cabin Embarked Fare Name Parch PassengerId Pclass Sex SibSp S 413 NaN NaN S 8.0500

Spector,

Mr. Woolf 0 1305 3 male 0

Oliva y
414 39.0 C105 C 108.9000 22.3583 Peter,
Master. Michael J
0 1306 1 female 0
Ocana, Dona.
415 38.5 NaN S 7.2500 Fermina

Saether, Mr. 0 1307 3 male 0

Simon
Sivertsen
416 NaN NaN S 8.0500 417 NaN NaN C
Ware,
Mr. 0 1308 3 male 0 1 1309 3 male 1
Frederick

Which features contain blank, null or empty values?

localhost:8888/notebooks/titanicMain (3).ipynb 6/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

These will require correcting.

Cabin > Age > Embarked features contain a number of null values in that order for the training dataset.
Cabin > Age are incomplete in case of test dataset.

What are the data types for various features?

Helping us during converting goal.

Seven features are integer or floats. Six in case of test dataset.

Five features are strings (object).

In [6]: 

train_df.info()
print('_'*40)
test_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId 891 non-null int64
 Survived 891 non-null int64
Pclass 891 non-null int64
Name 891 non-null object
Sex 891 non-null object
Age 714 non-null float64
SibSp 891 non-null int64
Parch 891 non-null int64
Ticket 891 non-null object
Fare 891 non-null float64
Cabin 204 non-null object
Embarked 889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB
________________________________________
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 418 entries, 0 to 417
Data columns (total 11 columns):
PassengerId 418 non-null int64
Pclass 418 non-null int64
Name 418 non-null object
Sex 418 non-null object
Age 332 non-null float64
SibSp 418 non-null int64
Parch 418 non-null int64
Ticket 418 non-null object
Fare 417 non-null float64
Cabin 91 non-null object
Embarked 418 non-null object
dtypes: float64(2), int64(4), object(5)
memory usage: 36.0+ KB

localhost:8888/notebooks/titanicMain (3).ipynb 7/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [10]: 

newdf.info()
print('_'*40)

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1309 entries, 0 to 417
Data columns (total 12 columns):
Age 1046 non-null float64
Cabin 295 non-null object
Embarked 1307 non-null object
Fare 1308 non-null float64
Name 1309 non-null object
Parch 1309 non-null int64
PassengerId 1309 non-null int64
Pclass 1309 non-null int64
Sex 1309 non-null object
SibSp 1309 non-null int64
Survived 891 non-null float64
Ticket 1309 non-null object
dtypes: float64(3), int64(4), object(5)
memory usage: 132.9+ KB
________________________________________
 What is the distribution of numerical feature values across the samples?

This helps us determine, among other early insights, how representative is the training dataset of the actual
problem domain.

Total samples are 891 or 40% of the actual number of passengers on board the Titanic
(2,224). Survived is a categorical feature with 0 or 1 values.
Around 38% samples survived representative of the actual survival rate at 32%.
Most passengers (> 75%) did not travel with parents or children.
Nearly 30% of the passengers had siblings and/or spouse aboard.
Fares varied significantly with few passengers (<1%) paying as high as $512.
Few elderly passengers (<1%) within age range 65-80.

localhost:8888/notebooks/titanicMain (3).ipynb 8/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [7]: 

train_df.describe()
# Review survived rate using `percentiles=[.61, .62]` knowing our problem description menti
# Review Parch distribution using `percentiles=[.75, .8]`
# SibSp distribution `[.68, .69]`
# Age and Fare `[.1, .2, .3, .4, .5, .6, .7, .8, .9, .99]`

Out[7]:

PassengerId Survived Pclass Age SibSp Parch Fare count 891.000000 891.000000

891.000000 714.000000 891.000000 891.000000 891.000000 mean 446.000000 0.383838 2.308642

29.699118 0.523008 0.381594 32.204208

std 257.353842 0.486592 0.836071 14.526497 1.102743 0.806057 49.693429 min 1.000000 0.000000

1.000000 0.420000 0.000000 0.000000 0.000000 25% 223.500000 0.000000 2.000000 20.125000

0.000000 0.000000 7.910400 50% 446.000000 0.000000 3.000000 28.000000 0.000000 0.000000

14.454200 75% 668.500000 1.000000 3.000000 38.000000 1.000000 0.000000 31.000000 max

891.000000 1.000000 3.000000 80.000000 8.000000 6.000000 512.329200

In [12]:  newdf.describe()
 Out[12]:

Age Fare Parch PassengerId Pclass SibSp Surviv

count 1046.000000 1308.000000 1309.000000 1309.000000 1309.000000 1309.000000 891.0000 mean

29.881138 33.295479 0.385027 655.000000 2.294882 0.498854 0.3838 std 14.413493 51.758668 0.865560

378.020061 0.837836 1.041658 0.4865 min 0.170000 0.000000 0.000000 1.000000 1.000000 0.000000 0.0000

25% 21.000000 7.895800 0.000000 328.000000 2.000000 0.000000 0.0000 50% 28.000000 14.454200

0.000000 655.000000 3.000000 0.000000 0.0000 75% 39.000000 31.275000 0.000000 982.000000 3.000000

1.000000 1.0000 max 80.000000 512.329200 9.000000 1309.000000 3.000000 8.000000 1.0000

What is the distribution of categorical features?

Names are unique across the dataset (count=unique=891)

Sex variable as two possible values with 65% male (top=male, freq=577/count=891).
Cabin values have several dupicates across samples. Alternatively several passengers shared a cabin.
Embarked takes three possible values. S port used by most passengers (top=S)
Ticket feature has high ratio (22%) of duplicate values (unique=681).

localhost:8888/notebooks/titanicMain (3).ipynb 9/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [8]:  train_df.describe(include=['O'])

Out[8]:

Name Sex Ticket Cabin Embarked

count 891 891 891 204 889

unique 891 2 681 147 3

top Goldsmith, Mr. Frank John male 347082 C23 C25 C27 S

freq 1 577 7 4 644

In [14]:  newdf.describe(include=['O'])

Out[14]:

Cabin Embarked Name Sex Ticket

count 295 1307 1309 1309 1309

unique 186 3 1307 2 929

top C23 C25 C27 S Kelly, Mr. James male CA. 2343

freq 6 914 2 843 11

Assumtions based on data analysis

 We arrive at following assumptions based on data analysis done so far. We may validate these assumptions
further before taking appropriate actions.

Correlating.

We want to know how well does each feature correlate with Survival. We want to do this early in our project and
match these quick correlations with modelled correlations later in the project.

Completing.

1. We may want to complete Age feature as it is definitely correlated to survival.

2. We may want to complete the Embarked feature as it may also correlate with survival or another important
feature.

Correcting.

1. Ticket feature may be dropped from our analysis as it contains high ratio of duplicates (22%) and there may
not be a correlation between Ticket and survival.
2. Cabin feature may be dropped as it is highly incomplete or contains many null values both in training and
test dataset.
3. PassengerId may be dropped from training dataset as it does not contribute to survival. 4. Name
feature is relatively non-standard, may not contribute directly to survival, so maybe dropped.

Creating.
localhost:8888/notebooks/titanicMain (3).ipynb 10/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

1. We may want to create a new feature called Family based on Parch and SibSp to get total count of family
members on board.
2. We may want to engineer the Name feature to extract Title as a new feature.
3. We may want to create new feature for Age bands. This turns a continous numerical feature into an
ordinal categorical feature.
4. We may also want to create a Fare range feature if it helps our analysis.

Classifying.

We may also add to our assumptions based on the problem description noted earlier.

1. Women (Sex=female) were more likely to have survived.

2. Children (Age<?) were more likely to have survived.
3. The upper-class passengers (Pclass=1) were more likely to have survived.

Analyze by pivoting features

To confirm some of our observations and assumptions, we can quickly analyze our feature correlations by
pivoting features against each other. We can only do so at this stage for features which do not have any empty
values. It also makes sense doing so only for features which are categorical (Sex), ordinal (Pclass) or discrete
(SibSp, Parch) type.

Pclass We observe significant correlation (>0.5) among Pclass=1 and Survived (classifying #3). We decide
to include this feature in our model.
Sex We confirm the observation during problem definition that Sex=female had very high survival rate at
74% (classifying #1).
SibSp and Parch These features have zero correlation for certain values. It may be best to derive a
feature or a set of features from these individual features (creating #1).
 In [9]:  train_df[['Pclass', 'Survived']].groupby(['Pclass'],

as_index=False).mean().sort_values(by=

Out[9]:

Pclass Survived

0 1 0.629630

1 2 0.472826

2 3 0.242363

localhost:8888/notebooks/titanicMain (3).ipynb 11/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [16]:  newdf[['Pclass', 'Survived']].groupby(['Pclass'],

as_index=False).mean().sort_values(by='Su

Out[16]:

Pclass Survived

0 1 0.629630

1 2 0.472826

2 3 0.242363

In [10]:  train_df[["Sex", "Survived"]].groupby(['Sex'],

as_index=False).mean().sort_values(by='Survi

Out[10]:

Sex Survived

0 female 0.742038

1 male 0.188908

In [18]:  newdf[["Sex", "Survived"]].groupby(['Sex'],

as_index=False).mean().sort_values(by='Survived

Out[18]:

Sex Survived

0 female 0.742038
 1 male 0.188908

In [11]:  train_df[["SibSp", "Survived"]].groupby(['SibSp'],

as_index=False).mean().sort_values(by='S

Out[11]:

SibSp Survived

1 1 0.535885

2 2 0.464286

0 0 0.345395

3 3 0.250000

4 4 0.166667

5 5 0.000000

6 8 0.000000

localhost:8888/notebooks/titanicMain (3).ipynb 12/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [20]:  newdf[["SibSp", "Survived"]].groupby(['SibSp'],

as_index=False).mean().sort_values(by='Surv

Out[20]:

SibSp Survived

1 1 0.535885

2 2 0.464286

0 0 0.345395

3 3 0.250000

4 4 0.166667

5 5 0.000000

6 8 0.000000

In [12]:  train_df[["Parch", "Survived"]].groupby(['Parch'],

as_index=False).mean().sort_values(by='S

Out[12]:

Parch Survived

3 3 0.600000

1 1 0.550847

2 2 0.500000

0 0 0.343658
 5 5 0.200000

4 4 0.000000

6 6 0.000000

localhost:8888/notebooks/titanicMain (3).ipynb 13/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [22]:  newdf[["Parch", "Survived"]].groupby(['Parch'],

as_index=False).mean().sort_values(by='Surv

Out[22]:

Parch Survived

3 3 0.600000

1 1 0.550847

2 2 0.500000

0 0 0.343658

5 5 0.200000

4 4 0.000000

6 6 0.000000

7 9 NaN

Analyze by visualizing data

Now we can continue confirming some of our assumptions using visualizations for analyzing the data.

Correlating numerical features

Let us start by understanding correlations between numerical features and our solution goal (Survived).

A histogram chart is useful for analyzing continous numerical variables like Age where banding or ranges will
help identify useful patterns. The histogram can indicate distribution of samples using automatically defined bins
or equally ranged bands. This helps us answer questions relating to specific bands (Did infants have better
survival rate?)
 Note that x-axis in historgram visualizations represents the count of samples or passengers.

Observations.

Infants (Age <=4) had high survival rate.

Oldest passengers (Age = 80) survived.
Large number of 15-25 year olds did not survive.
Most passengers are in 15-35 age range.

Decisions.

This simple analysis confirms our assumptions as decisions for subsequent workflow stages.

We should consider Age (our assumption classifying #2) in our model training.
Complete the Age feature for null values (completing #1).
We should band age groups (creating #3).

localhost:8888/notebooks/titanicMain (3).ipynb 14/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [13]: 

g = sns.FacetGrid(train_df, col='Survived')
g.map(plt.hist, 'Age', bins=20)

Out[13]:

<seaborn.axisgrid.FacetGrid at 0x2a37db59b70>

In [24]: 

g = sns.FacetGrid(newdf, col='Survived')
g.map(plt.hist, 'Age', bins=20)

Out[24]:

<seaborn.axisgrid.FacetGrid at 0x2c61f930a90>
 Correlating numerical and ordinal features
We can combine multiple features for identifying correlations using a single plot. This can be done with
numerical and categorical features which have numeric values.

Observations.

Pclass=3 had most passengers, however most did not survive. Confirms our classifying assumption #2.
localhost:8888/notebooks/titanicMain (3).ipynb 15/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

Infant passengers in Pclass=2 and Pclass=3 mostly survived. Further qualifies our classifying assumption
#2.
Most passengers in Pclass=1 survived. Confirms our classifying assumption #3.
Pclass varies in terms of Age distribution of passengers.

Decisions.

Consider Pclass for model training.

In [14]: 

# grid = sns.FacetGrid(train_df, col='Pclass', hue='Survived')

grid = sns.FacetGrid(train_df, col='Survived', row='Pclass', size=2.2, aspect=1.6)
grid.map(plt.hist, 'Age', alpha=.5, bins=20)
grid.add_legend();
 localhost:8888/notebooks/titanicMain (3).ipynb 16/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [26]: 

# grid = sns.FacetGrid(train_df, col='Pclass', hue='Survived')

grid = sns.FacetGrid(newdf, col='Survived', row='Pclass', size=2.2, aspect=1.6)
grid.map(plt.hist, 'Age', alpha=.5, bins=20)
grid.add_legend();
 Correlating categorical features
Now we can correlate categorical features with our solution goal.

Observations.

Female passengers had much better survival rate than males. Confirms classifying (#1). Exception in
Embarked=C where males had higher survival rate. This could be a correlation between Pclass and
Embarked and in turn Pclass and Survived, not necessarily direct correlation between
localhost:8888/notebooks/titanicMain (3).ipynb 17/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook
c ass a d ba ed a d tu c ass a d Su ed, ot ecessa y d ect co e at o bet ee Embarked and
Survived.
Males had better survival rate in Pclass=3 when compared with Pclass=2 for C and Q ports. Completing
(#2).
Ports of embarkation have varying survival rates for Pclass=3 and among male passengers. Correlating
(#1).

Decisions.

Add Sex feature to model training.

Complete and add Embarked feature to model training.
 In [15]: 

# grid = sns.FacetGrid(train_df, col='Embarked')

grid = sns.FacetGrid(train_df, row='Embarked', size=2.2, aspect=1.6)
grid.map(sns.pointplot, 'Pclass', 'Survived', 'Sex', palette='deep')
grid.add_legend()

Out[15]:

<seaborn.axisgrid.FacetGrid at 0x2a37e6febe0>

localhost:8888/notebooks/titanicMain (3).ipynb 18/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [28]: 

# grid = sns.FacetGrid(train_df, col='Embarked')

grid = sns.FacetGrid(newdf, row='Embarked', size=2.2, aspect=1.6)
grid.map(sns.pointplot, 'Pclass', 'Survived', 'Sex', palette='deep')
grid.add_legend()

Out[28]:

<seaborn.axisgrid.FacetGrid at 0x2c620547cc0>
 Correlating categorical and numerical features
We may also want to correlate categorical features (with non-numeric values) and numeric features. We can
consider correlating Embarked (Categorical non-numeric), Sex (Categorical non-numeric), Fare (Numeric
continuous), with Survived (Categorical numeric).

Observations.

Higher fare paying passengers had better survival. Confirms our assumption for creating (#4) fare
ranges. Port of embarkation correlates with survival rates. Confirms correlating (#1) and completing (#2).

localhost:8888/notebooks/titanicMain (3).ipynb 19/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

Decisions.

Consider banding Fare feature.

In [16]: 

# grid = sns.FacetGrid(train_df, col='Embarked', hue='Survived', palette={0: 'k', 1: 'w'})

grid = sns.FacetGrid(train_df, row='Embarked', col='Survived', size=2.2, aspect=1.6)
grid.map(sns.barplot, 'Sex', 'Fare', alpha=.5, ci=None)
grid.add_legend()

Out[16]:

<seaborn.axisgrid.FacetGrid at 0x2a37e6fe8d0>
 localhost:8888/notebooks/titanicMain (3).ipynb 20/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [30]: 

# grid = sns.FacetGrid(train_df, col='Embarked', hue='Survived', palette={0: 'k', 1: 'w'})

grid = sns.FacetGrid(newdf, row='Embarked', col='Survived', size=2.2, aspect=1.6)
grid.map(sns.barplot, 'Sex', 'Fare', alpha=.5, ci=None)
grid.add_legend()

Out[30]:

<seaborn.axisgrid.FacetGrid at 0x2c6206518d0>
 Wrangle data

localhost:8888/notebooks/titanicMain (3).ipynb 21/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

We have collected several assumptions and decisions regarding our datasets and solution requirements. So far
we did not have to change a single feature or value to arrive at these. Let us now execute our decisions and
assumptions for correcting, creating, and completing goals.

Correcting by dropping features

This is a good starting goal to execute. By dropping features we are dealing with fewer data points. Speeds up
our notebook and eases the analysis.

Based on our assumptions and decisions we want to drop the Cabin (correcting #2) and Ticket (correcting #1)
features.

Note that where applicable we perform operations on both training and testing datasets together to stay
consistent.
 In [17]:  print("Before", train_df.shape, test_df.shape, combine[0].shape,
combine[1].shape)

train_df = train_df.drop(['Ticket', 'Cabin'], axis=1)

test_df = test_df.drop(['Ticket', 'Cabin'], axis=1)
combine = [train_df, test_df]

"After", train_df.shape, test_df.shape, combine[0].shape, combine[1].shape

Before (891, 12) (418, 11) (891, 12) (418, 11)

Out[17]:

('After', (891, 10), (418, 9), (891, 10), (418, 9))

In [32]:  print("Before", newdf.shape)

newdf = newdf.drop(['Ticket', 'Cabin'], axis=1)

"After", newdf.shape

Before (1309, 12)

Out[32]:

('After', (1309, 10))

Creating new feature extracting from existing

We want to analyze if Name feature can be engineered to extract titles and test correlation between titles and
survival, before dropping Name and PassengerId features.

In the following code we extract Title feature using regular expressions. The RegEx pattern (\w+\.) matches
the first word which ends with a dot character within Name feature. The expand=False flag returns a
DataFrame.

Observations.

localhost:8888/notebooks/titanicMain (3).ipynb 22/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

When we plot Title, Age, and Survived, we note the following observations.

Most titles band Age groups accurately. For example: Master title has Age mean of 5
years. Survival among Title Age bands varies slightly.
Certain titles mostly survived (Mme, Lady, Sir) or did not (Don, Rev, Jonkheer).

Decision.

We decide to retain the new Title feature for model training.

In [18]: 

for dataset in combine:

dataset['Title'] = dataset.Name.str.extract(' ([A-Za-z]+)\.', expand=False)

pd.crosstab(train_df['Title'], train_df['Sex'])
 Out[18]:

Sex female male

Title

Capt 0 1

Col 0 2

Countess 1 0

Don 0 1

Dr 1 6

Jonkheer 0 1

Lady 1 0

Major 0 2

Master 0 40

Miss 182 0

Mlle 2 0

Mme 1 0

Mr 0 517

Mrs 125 0

Ms 1 0

Rev 0 6

Sir 0 1

localhost:8888/notebooks/titanicMain (3).ipynb 23/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [34]: 

combine1 = [newdf] # this is to convert dataframe beome array

for dataset in combine1:
dataset['Title'] = dataset.Name.str.extract(' ([A-Za-z]+)\.', expand=False)

pd.crosstab(newdf['Title'], newdf['Sex'])

Out[34]:

Sex female male

Title

Capt 0 1

Col 0 4

Countess 1 0
 Don 0 1

Dona 1 0

Dr 1 7

Jonkheer 0 1

Lady 1 0

Major 0 2

Master 0 61

Miss 260 0

Mlle 2 0

Mme 1 0

Mr 0 757

Mrs 197 0

Ms 2 0

Rev 0 8

Sir 0 1

localhost:8888/notebooks/titanicMain (3).ipynb 24/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [19]:  pd.crosstab(train_df['Title'], train_df['Survived'])

Out[19]:

Survived 0 1

Title

Capt 1 0

Col 1 1

Countess 0 1

Don 1 0

Dr 4 3

Jonkheer 1 0

Lady 0 1
 Major 1 1

Master 17 23

Miss 55 127

Mlle 0 2

Mme 0 1

Mr 436 81

Mrs 26 99

Ms 0 1

Rev 6 0

Sir 0 1

localhost:8888/notebooks/titanicMain (3).ipynb 25/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [36]:  pd.crosstab(newdf['Title'], newdf['Survived'])

Out[36]:

Survived 0.0 1.0

Title

Capt 1 0

Col 1 1

Countess 0 1

Don 1 0

Dr 4 3

Jonkheer 1 0

Lady 0 1

Major 1 1
 Master 17 23

Miss 55 127

Mlle 0 2

Mme 0 1

Mr 436 81

Mrs 26 99

Ms 0 1

Rev 6 0

Sir 0 1

We can replace many titles with a more common name or classify them as Rare .

localhost:8888/notebooks/titanicMain (3).ipynb 26/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [20]: 

for dataset in combine:

dataset['Title'] = dataset['Title'].replace(['Lady', 'Countess','Capt', 'Col',\
'Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')

dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')

dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')
dataset['Title'] = dataset['Title'].replace('Mme', 'Mrs')

train_df[['Title', 'Survived']].groupby(['Title'], as_index=False).mean()

Out[20]:

Title Survived

0 Master 0.575000

1 Miss 0.702703

2 Mr 0.156673

3 Mrs 0.793651
 4 Rare 0.347826

In [38]: 

for dataset in combine1:

dataset['Title'] = dataset['Title'].replace(['Lady', 'Countess','Capt', 'Col',\
'Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')

dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')

dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')
dataset['Title'] = dataset['Title'].replace('Mme', 'Mrs')

newdf[['Title', 'Survived']].groupby(['Title'], as_index=False).mean()

Out[38]:

Title Survived

0 Master 0.575000

1 Miss 0.702703

2 Mr 0.156673

3 Mrs 0.793651

4 Rare 0.347826

We can convert the categorical titles to ordinal.

localhost:8888/notebooks/titanicMain (3).ipynb 27/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [21]: 

title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5}

for dataset in combine:
dataset['Title'] = dataset['Title'].map(title_mapping)
dataset['Title'] = dataset['Title'].fillna(0)

train_df.head()

Out[21]:

PassengerId Survived Pclass Name Sex Age SibSp Parch Fare Embarked Braund,

01031211231334114503 Heikkinen, Miss. 71.2833 C female 26.0 0 0 7.9250 S female

Laina

Futrelle,
Mrs.
Mr. Owen Harris Jacques Heath 35.0 1 0 53.1000 S male 35.0 0 0 8.0500 S
(Lily May Peel)
Cumings, Mrs. John Bradley
(Florence Briggs Allen, Mr. William
Th... Henry
male 22.0 1 0 7.2500 S female 38.0 1 0
 localhost:8888/notebooks/titanicMain (3).ipynb 28/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [40]: 

title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5}

for dataset in combine1:
dataset['Title'] = dataset['Title'].map(title_mapping)
dataset['Title'] = dataset['Title'].fillna(0)

newdf.head()

Out[40]:

Age Embarked Fare Name Parch PassengerId Pclass Sex SibSp Survived Braund,

0 22.0 S 7.2500 1 38.0 C 71.2833 2 26.0 S Cumings, Mrs. John Bradley Henry
(Florence Briggs 0 1 3 male 1 0.0 0 2 1 female 1 1.0 0 3 3
Th...

Heikkinen, Miss.
7.9250 3 35.0 S 53.1000 4 35.0 S 8.0500 Laina
female 0 1.0 0 4 1 female 1 1.0 0 5 3 male 0
Futrelle,
Mrs.
Jacques Heath
(Lily May Peel)
Mr. Owen Harris
Allen, Mr. William 0.0

Now we can safely drop the Name feature from training and testing datasets. We also do not need the
PassengerId feature in the training dataset.

In [22]: 

train_df = train_df.drop(['Name', 'PassengerId'], axis=1)

test_df = test_df.drop(['Name'], axis=1)
combine = [train_df, test_df]
train_df.shape, test_df.shape

Out[22]:

((891, 9), (418, 9))

localhost:8888/notebooks/titanicMain (3).ipynb 29/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [42]: 

newdf = newdf.drop(['Name', 'PassengerId'], axis=1)

combine1 = [newdf]
newdf.shape
 Out[42]:

(1309, 9)

In [43]:  newdf.head()

Out[43]:

Age Embarked Fare Parch Pclass Sex SibSp Survived Title

0 22.0 S 7.2500 0 3 male 1 0.0 1

1 38.0 C 71.2833 0 1 female 1 1.0 3

2 26.0 S 7.9250 0 3 female 0 1.0 2

3 35.0 S 53.1000 0 1 female 1 1.0 3

4 35.0 S 8.0500 0 3 male 0 0.0 1

Converting a categorical feature

Now we can convert features which contain strings to numerical values. This is required by most
model algorithms. Doing so will also help us in achieving the feature completing goal.

Let us start by converting Sex feature to a new feature called Gender where female=1 and male=0.

In [23]: 

for dataset in combine:

dataset['Sex'] = dataset['Sex'].map( {'female': 1, 'male': 0} ).astype(int)

train_df.head()

Out[23]:

Survived Pclass Sex Age SibSp Parch Fare Embarked Title

0 0 3 0 22.0 1 0 7.2500 S 1

1 1 1 1 38.0 1 0 71.2833 C 3

2 1 3 1 26.0 0 0 7.9250 S 2

3 1 1 1 35.0 1 0 53.1000 S 3

4 0 3 0 35.0 0 0 8.0500 S 1

localhost:8888/notebooks/titanicMain (3).ipynb 30/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [45]: 

for dataset in combine1:

dataset['Sex'] = dataset['Sex'].map( {'female': 1, 'male': 0} ).astype(int)

newdf.head()
 Out[45]:

Age Embarked Fare Parch Pclass Sex SibSp Survived Title

0 22.0 S 7.2500 0 3 0 1 0.0 1

1 38.0 C 71.2833 0 1 1 1 1.0 3

2 26.0 S 7.9250 0 3 1 0 1.0 2

3 35.0 S 53.1000 0 1 1 1 1.0 3

4 35.0 S 8.0500 0 3 0 0 0.0 1

Completing a numerical continuous feature

Now we should start estimating and completing features with missing or null values. We will first do this for the
Age feature.

We can consider three methods to complete a numerical continuous feature.

1. A simple way is to generate random numbers between mean and standard deviation
(https://en.wikipedia.org/wiki/Standard_deviation).
2. More accurate way of guessing missing values is to use other correlated features. In our case we note
correlation among Age, Gender, and Pclass. Guess Age values using median
(https://en.wikipedia.org/wiki/Median) values for Age across sets of Pclass and Gender feature
combinations. So, median Age for Pclass=1 and Gender=0, Pclass=1 and Gender=1, and so on... 3.
Combine methods 1 and 2. So instead of guessing age values based on median, use random numbers
between mean and standard deviation, based on sets of Pclass and Gender combinations.

Method 1 and 3 will introduce random noise into our models. The results from multiple executions might
vary. We will prefer method 2.

localhost:8888/notebooks/titanicMain (3).ipynb 31/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [24]: 

# grid = sns.FacetGrid(train_df, col='Pclass', hue='Gender')

grid = sns.FacetGrid(train_df, row='Pclass', col='Sex', size=2.2, aspect=1.6)
grid.map(plt.hist, 'Age', alpha=.5, bins=20)
grid.add_legend()

Out[24]:
 <seaborn.axisgrid.FacetGrid at 0x2a37ec71390>

localhost:8888/notebooks/titanicMain (3).ipynb 32/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [47]: 

# grid = sns.FacetGrid(train_df, col='Pclass', hue='Gender')

grid = sns.FacetGrid(newdf, row='Pclass', col='Sex', size=2.2, aspect=1.6)
grid.map(plt.hist, 'Age', alpha=.5, bins=20)
grid.add_legend()

Out[47]:

<seaborn.axisgrid.FacetGrid at 0x2c621e22f60>
 localhost:8888/notebooks/titanicMain (3).ipynb 33/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [25]: 

# KNN mean computation method

#try to get the mean age according to particular pclass , sex and embarked
#female=1 and male=0.
train_age= train_df[["Age", "Sex", "Pclass", "Embarked"]].groupby(['Pclass','Embarked', 'Se
train_age

Out[25]:

Pclass Embarked Sex Age

0 1 C 0 40.111111

1 1 C 1 36.052632

2 1 Q 0 44.000000
 3 1 Q 1 33.000000

4 1 S 0 41.897188

5 1 S 1 32.704545

11 2 S 1 29.719697

10 2 S 0 30.875889

9 2 Q 1 30.000000

8 2 Q 0 57.000000

7 2 C 1 19.142857

6 2 C 0 25.937500

12 3 C 0 25.016800

13 3 C 1 14.062500

14 3 Q 0 28.142857

15 3 Q 1 22.850000

16 3 S 0 26.574766

17 3 S 1 23.223684

localhost:8888/notebooks/titanicMain (3).ipynb 34/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [49]: 

# KNN mean computation method

#try to get the mean age according to particular pclass , sex and embarked
#female=1 and male=0.
train_age= newdf[["Age", "Sex", "Pclass", "Embarked"]].groupby(['Pclass','Embarked', 'Sex']
train_age

Out[49]:

Pclass Embarked Sex Age

0 1 C 0 40.047619

1 1 C 1 38.107692

2 1 Q 0 44.000000
 3 1 Q 1 35.000000

4 1 S 0 41.705977

5 1 S 1 35.609375

11 2 S 1 28.455165

10 2 S 0 30.491702

9 2 Q 1 30.000000

8 2 Q 0 53.750000

7 2 C 1 19.363636

6 2 C 0 27.269231

12 3 C 0 24.129474

13 3 C 1 16.818182

14 3 Q 0 26.738095

15 3 Q 1 24.333333

16 3 S 0 26.146241

17 3 S 1 22.854771

Let us start by preparing an empty array to contain guessed Age values based on Pclass x Gender
combinations.

In [26]: 

guess_ages = np.zeros((2,3))
guess_ages

Out[26]:

array([[ 0., 0., 0.],

[ 0., 0., 0.]])

Now we iterate over Sex (0 or 1) and Pclass (1, 2, 3) to calculate guessed values of Age for the six

localhost:8888/notebooks/titanicMain (3).ipynb 35/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

combinations.

In [27]: 

for dataset in combine:

for i in range(0, 2):
for j in range(0, 3):
guess_df = dataset[(dataset['Sex'] == i) & \
(dataset['Pclass'] == j+1)]['Age'].dropna()

# age_mean = guess_df.mean()
# age_std = guess_df.std()
# age_guess = rnd.uniform(age_mean - age_std, age_mean + age_std) age_guess

= guess_df.median()

# Convert random age float to nearest .5 age

 guess_ages[i,j] = int( age_guess/0.5 + 0.5 ) * 0.5

for i in range(0, 2):

for j in range(0, 3):
dataset.loc[ (dataset.Age.isnull()) & (dataset.Sex == i) & (dataset.Pclass == j 'Age'] =
guess_ages[i,j]

dataset['Age'] = dataset['Age'].astype(int)

train_df.head()

Out[27]:

Survived Pclass Sex Age SibSp Parch Fare Embarked Title

0 0 3 0 22 1 0 7.2500 S 1

1 1 1 1 38 1 0 71.2833 C 3

2 1 3 1 26 0 0 7.9250 S 2

3 1 1 1 35 1 0 53.1000 S 3

4 0 3 0 35 0 0 8.0500 S 1

localhost:8888/notebooks/titanicMain (3).ipynb 36/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [52]: 

for dataset in combine1:

for i in range(0, 2):
for j in range(0, 3):
guess_df = dataset[(dataset['Sex'] == i) & \
(dataset['Pclass'] == j+1)]['Age'].dropna()

# age_mean = guess_df.mean()
# age_std = guess_df.std()
# age_guess = rnd.uniform(age_mean - age_std, age_mean + age_std) age_guess

= guess_df.median()

# Convert random age float to nearest .5 age

guess_ages[i,j] = int( age_guess/0.5 + 0.5 ) * 0.5
 for i in range(0, 2):
for j in range(0, 3):
dataset.loc[ (dataset.Age.isnull()) & (dataset.Sex == i) & (dataset.Pclass == j 'Age'] =
guess_ages[i,j]

dataset['Age'] = dataset['Age'].astype(int)

newdf.head()

Out[52]:

Age Embarked Fare Parch Pclass Sex SibSp Survived Title

0 22 S 7.2500 0 3 0 1 0.0 1

1 38 C 71.2833 0 1 1 1 1.0 3

2 26 S 7.9250 0 3 1 0 1.0 2

3 35 S 53.1000 0 1 1 1 1.0 3

4 35 S 8.0500 0 3 0 0 0.0 1

Let us create Age bands and determine correlations with Survived.

In [28]: 

train_df['AgeBand'] = pd.cut(train_df['Age'], 5)
train_df[['AgeBand', 'Survived']].groupby(['AgeBand'], as_index=False).mean().sort_values(b

Out[28]:

AgeBand Survived

0 (-0.08, 16.0] 0.550000

1 (16.0, 32.0] 0.337374

2 (32.0, 48.0] 0.412037

3 (48.0, 64.0] 0.434783

4 (64.0, 80.0] 0.090909

localhost:8888/notebooks/titanicMain (3).ipynb 37/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [54]: 

newdf['AgeBand'] = pd.cut(newdf['Age'], 5)
newdf[['AgeBand', 'Survived']].groupby(['AgeBand'], as_index=False).mean().sort_values(by='

Out[54]:

AgeBand Survived

0 (-0.08, 16.0] 0.550000

1 (16.0, 32.0] 0.337374

2 (32.0, 48.0] 0.412037

3 (48.0, 64.0] 0.434783

4 (64.0, 80.0] 0.090909

 Let us replace Age with ordinals based on these bands.

In [29]: 

for dataset in combine:

dataset.loc[ dataset['Age'] <= 16, 'Age'] = 0
dataset.loc[(dataset['Age'] > 16) & (dataset['Age'] <= 32), 'Age'] = 1
dataset.loc[(dataset['Age'] > 32) & (dataset['Age'] <= 48), 'Age'] = 2
dataset.loc[(dataset['Age'] > 48) & (dataset['Age'] <= 64), 'Age'] = 3
dataset.loc[ dataset['Age'] > 64, 'Age']
train_df.head()

Out[29]:

Survived Pclass Sex Age SibSp Parch Fare Embarked Title AgeBand

0 0 3 0 1 1 0 7.2500 S 1 (16.0, 32.0]

1 1 1 1 2 1 0 71.2833 C 3 (32.0, 48.0]

2 1 3 1 1 0 0 7.9250 S 2 (16.0, 32.0]

3 1 1 1 2 1 0 53.1000 S 3 (32.0, 48.0]

4 0 3 0 2 0 0 8.0500 S 1 (32.0, 48.0]

localhost:8888/notebooks/titanicMain (3).ipynb 38/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [56]: 

for dataset in combine1:

dataset.loc[ dataset['Age'] <= 16, 'Age'] = 0
dataset.loc[(dataset['Age'] > 16) & (dataset['Age'] <= 32), 'Age'] = 1
dataset.loc[(dataset['Age'] > 32) & (dataset['Age'] <= 48), 'Age'] = 2
dataset.loc[(dataset['Age'] > 48) & (dataset['Age'] <= 64), 'Age'] = 3
dataset.loc[ dataset['Age'] > 64, 'Age']
newdf.head()

Out[56]:

Age Embarked Fare Parch Pclass Sex SibSp Survived Title AgeBand

0 1 S 7.2500 0 3 0 1 0.0 1 (16.0, 32.0]

1 2 C 71.2833 0 1 1 1 1.0 3 (32.0, 48.0]

2 1 S 7.9250 0 3 1 0 1.0 2 (16.0, 32.0]

3 2 S 53.1000 0 1 1 1 1.0 3 (32.0, 48.0]

 4 2 S 8.0500 0 3 0 0 0.0 1 (32.0, 48.0]

We can not remove the AgeBand feature.

In [30]: 

train_df = train_df.drop(['AgeBand'], axis=1)

combine = [train_df, test_df]
train_df.head()

Out[30]:

Survived Pclass Sex Age SibSp Parch Fare Embarked Title

0 0 3 0 1 1 0 7.2500 S 1

1 1 1 1 2 1 0 71.2833 C 3

2 1 3 1 1 0 0 7.9250 S 2

3 1 1 1 2 1 0 53.1000 S 3

4 0 3 0 2 0 0 8.0500 S 1

localhost:8888/notebooks/titanicMain (3).ipynb 39/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [58]: 

newdf = newdf.drop(['AgeBand'], axis=1)

combine1 = [newdf]
newdf.head()

Out[58]:

Age Embarked Fare Parch Pclass Sex SibSp Survived Title

0 1 S 7.2500 0 3 0 1 0.0 1

1 2 C 71.2833 0 1 1 1 1.0 3

2 1 S 7.9250 0 3 1 0 1.0 2

3 2 S 53.1000 0 1 1 1 1.0 3

4 2 S 8.0500 0 3 0 0 0.0 1
 Create new feature combining existing features
We can create a new feature for FamilySize which combines Parch and SibSp. This will enable us to drop
Parch and SibSp from our datasets.

In [31]: 

for dataset in combine:

dataset['FamilySize'] = dataset['SibSp'] + dataset['Parch'] + 1

train_df[['FamilySize', 'Survived']].groupby(['FamilySize'], as_index=False).mean().sort_va

Out[31]:

FamilySize Survived

3 4 0.724138

2 3 0.578431

1 2 0.552795

6 7 0.333333

0 1 0.303538

4 5 0.200000

5 6 0.136364

7 8 0.000000

8 11 0.000000

localhost:8888/notebooks/titanicMain (3).ipynb 40/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [60]: 

for dataset in combine1:

dataset['FamilySize'] = dataset['SibSp'] + dataset['Parch'] + 1

newdf[['FamilySize', 'Survived']].groupby(['FamilySize'], as_index=False).mean().sort_value

Out[60]:

FamilySize Survived

3 4 0.724138

2 3 0.578431

1 2 0.552795

6 7 0.333333

0 1 0.303538

4 5 0.200000
 5 6 0.136364

7 8 0.000000

8 11 0.000000

We can create another feature called IsAlone.

In [32]: 

for dataset in combine:

dataset['IsAlone'] = 0
dataset.loc[dataset['FamilySize'] == 1, 'IsAlone'] = 1

train_df[['IsAlone', 'Survived']].groupby(['IsAlone'], as_index=False).mean()

Out[32]:

IsAlone Survived

0 0 0.505650

1 1 0.303538

localhost:8888/notebooks/titanicMain (3).ipynb 41/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [62]: 

for dataset in combine1:

dataset['IsAlone'] = 0
dataset.loc[dataset['FamilySize'] == 1, 'IsAlone'] = 1

newdf[['IsAlone', 'Survived']].groupby(['IsAlone'], as_index=False).mean()

Out[62]:

IsAlone Survived

0 0 0.505650

1 1 0.303538

Let us drop Parch, SibSp, and FamilySize features in favor of IsAlone.

In [33]: 
 train_df = train_df.drop(['Parch', 'SibSp', 'FamilySize'], axis=1)
test_df = test_df.drop(['Parch', 'SibSp', 'FamilySize'], axis=1)
combine = [train_df, test_df]

train_df.head()

Out[33]:

Survived Pclass Sex Age Fare Embarked Title IsAlone

0 0 3 0 1 7.2500 S 1 0

1 1 1 1 2 71.2833 C 3 0

2 1 3 1 1 7.9250 S 2 1

3 1 1 1 2 53.1000 S 3 0

4 0 3 0 2 8.0500 S 1 1

In [64]: 

newdf = newdf.drop(['Parch', 'SibSp', 'FamilySize'], axis=1)

combine1 = [newdf]

newdf.head()

Out[64]:

Age Embarked Fare Pclass Sex Survived Title IsAlone

0 1 S 7.2500 3 0 0.0 1 0

1 2 C 71.2833 1 1 1.0 3 0

2 1 S 7.9250 3 1 1.0 2 1

3 2 S 53.1000 1 1 1.0 3 0

4 2 S 8.0500 3 0 0.0 1 1

We can also create an artificial feature combining Pclass and Age.

localhost:8888/notebooks/titanicMain (3).ipynb 42/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [34]: 

for dataset in combine:

dataset['Age*Class'] = dataset.Age * dataset.Pclass

train_df.loc[:, ['Age*Class', 'Age', 'Pclass']].head(10)

Out[34]:

Age*Class Age Pclass

0313

1221

2313

3221

4623

5313
 6331

7003

8313

9002

In [66]: 

for dataset in combine1:

dataset['Age*Class'] = dataset.Age * dataset.Pclass

newdf.loc[:, ['Age*Class', 'Age', 'Pclass']].head(10)

Out[66]:

Age*Class Age Pclass

0313

1221

2313

3221

4623

5313

6331

7003

8313

9002

Completing a categorical feature

Embarked feature takes S, Q, C values based on port of embarkation. Our training dataset has two missing
values. We simply fill these with the most common occurance.

localhost:8888/notebooks/titanicMain (3).ipynb 43/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [35]: 

freq_port = train_df.Embarked.dropna().mode()[0]
freq_port

Out[35]:

'S'

In [68]: 

freq_port = newdf.Embarked.dropna().mode()[0]
freq_port

Out[68]:

'S'

In [36]: 
 for dataset in combine:
dataset['Embarked'] = dataset['Embarked'].fillna(freq_port)

train_df[['Embarked', 'Survived']].groupby(['Embarked'], as_index=False).mean().sort_values

Out[36]:

Embarked Survived

0 C 0.553571

1 Q 0.389610

2 S 0.339009

In [70]: 

for dataset in combine1:

dataset['Embarked'] = dataset['Embarked'].fillna(freq_port)

newdf[['Embarked', 'Survived']].groupby(['Embarked'], as_index=False).mean().sort_values(by

Out[70]:

Embarked Survived

0 C 0.553571

1 Q 0.389610

2 S 0.339009

Converting categorical feature to numeric

We can now convert the EmbarkedFill feature by creating a new numeric Port feature.

localhost:8888/notebooks/titanicMain (3).ipynb 44/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [37]: 

for dataset in combine:

dataset['Embarked'] = dataset['Embarked'].map( {'S': 0, 'C': 1, 'Q': 2} ).astype(int)

train_df.head()

Out[37]:

Survived Pclass Sex Age Fare Embarked Title IsAlone Age*Class

0 0 3 0 1 7.2500 0 1 0 3

1 1 1 1 2 71.2833 1 3 0 2

2 1 3 1 1 7.9250 0 2 1 3

3 1 1 1 2 53.1000 0 3 0 2

4 0 3 0 2 8.0500 0 1 1 6
 In [72]: 

for dataset in combine1:

dataset['Embarked'] = dataset['Embarked'].map( {'S': 0, 'C': 1, 'Q': 2} ).astype(int)

newdf.head()

Out[72]:

Age Embarked Fare Pclass Sex Survived Title IsAlone Age*Class

0 1 0 7.2500 3 0 0.0 1 0 3

1 2 1 71.2833 1 1 1.0 3 0 2

2 1 0 7.9250 3 1 1.0 2 1 3

3 2 0 53.1000 1 1 1.0 3 0 2

4 2 0 8.0500 3 0 0.0 1 1 6

Quick completing and converting a numeric feature

We can now complete the Fare feature for single missing value in test dataset using mode to get the value that
occurs most frequently for this feature. We do this in a single line of code.

Note that we are not creating an intermediate new feature or doing any further analysis for correlation to guess
missing feature as we are replacing only a single value. The completion goal achieves desired requirement for
model algorithm to operate on non-null values.

We may also want round off the fare to two decimals as it represents currency.

localhost:8888/notebooks/titanicMain (3).ipynb 45/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [38]: 

test_df['Fare'].fillna(test_df['Fare'].dropna().median(), inplace=True)
test_df.head()

Out[38]:

PassengerId Pclass Sex Age Fare Embarked Title IsAlone Age*Class

0 892 3 0 2 7.8292 2 1 1 6

1 893 3 1 2 7.0000 0 3 0 6

2 894 2 0 3 9.6875 2 1 1 6

3 895 3 0 1 8.6625 0 1 1 3

4 896 3 1 1 12.2875 0 3 0 3

In [74]: 

newdf['Fare'].fillna(newdf['Fare'].dropna().median(), inplace=True)
newdf.head()
 Out[74]:

Age Embarked Fare Pclass Sex Survived Title IsAlone Age*Class

0 1 0 7.2500 3 0 0.0 1 0 3

1 2 1 71.2833 1 1 1.0 3 0 2

2 1 0 7.9250 3 1 1.0 2 1 3

3 2 0 53.1000 1 1 1.0 3 0 2

4 2 0 8.0500 3 0 0.0 1 1 6

We can not create FareBand.

In [39]: 

train_df['FareBand'] = pd.qcut(train_df['Fare'], 4)
train_df[['FareBand', 'Survived']].groupby(['FareBand'], as_index=False).mean().sort_values

Out[39]:

FareBand Survived

0 (-0.001, 7.91] 0.197309

1 (7.91, 14.454] 0.303571

2 (14.454, 31.0] 0.454955

3 (31.0, 512.329] 0.581081

localhost:8888/notebooks/titanicMain (3).ipynb 46/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [76]: 

newdf['FareBand'] = pd.qcut(newdf['Fare'], 4)
newdf[['FareBand', 'Survived']].groupby(['FareBand'], as_index=False).mean().sort_values(by

Out[76]:

FareBand Survived

0 (-0.001, 7.896] 0.197309

1 (7.896, 14.454] 0.303571

2 (14.454, 31.275] 0.441048

3 (31.275, 512.329] 0.600000

Convert the Fare feature to ordinal values based on the FareBand.

In [40]: 
 for dataset in combine:
dataset.loc[ dataset['Fare'] <= 7.91, 'Fare'] = 0
dataset.loc[(dataset['Fare'] > 7.91) & (dataset['Fare'] <= 14.454), 'Fare'] = 1
dataset.loc[(dataset['Fare'] > 14.454) & (dataset['Fare'] <= 31), 'Fare'] = 2
dataset.loc[ dataset['Fare'] > 31, 'Fare'] = 3
dataset['Fare'] = dataset['Fare'].astype(int)

train_df = train_df.drop(['FareBand'], axis=1)

combine = [train_df, test_df]

train_df.head(10)

Out[40]:

Survived Pclass Sex Age Fare Embarked Title IsAlone Age*Class

0030100103

1111231302

2131110213

3111230302

4030210116

5030112113

6010330113

7030020400

8131110303

9121021300

localhost:8888/notebooks/titanicMain (3).ipynb 47/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [78]: 

for dataset in combine1:

dataset.loc[ dataset['Fare'] <= 7.91, 'Fare'] = 0
dataset.loc[(dataset['Fare'] > 7.91) & (dataset['Fare'] <= 14.454), 'Fare'] = 1
dataset.loc[(dataset['Fare'] > 14.454) & (dataset['Fare'] <= 31), 'Fare'] = 2
dataset.loc[ dataset['Fare'] > 31, 'Fare'] = 3
dataset['Fare'] = dataset['Fare'].astype(int)

newdf = newdf.drop(['FareBand'], axis=1)

combine1 = [newdf]

newdf.head(10)

Out[78]:

Age Embarked Fare Pclass Sex Survived Title IsAlone Age*Class

0 1 0 0 3 0 0.0 1 0 3

1 2 1 3 1 1 1.0 3 0 2

2 1 0 1 3 1 1.0 2 1 3
 3 2 0 3 1 1 1.0 3 0 2

4 2 0 1 3 0 0.0 1 1 6

5 1 2 1 3 0 0.0 1 1 3

6 3 0 3 1 0 0.0 1 1 3

7 0 0 2 3 0 0.0 4 0 0

8 1 0 1 3 1 1.0 3 0 3

9 0 1 2 2 1 1.0 3 0 0

And the test dataset.

localhost:8888/notebooks/titanicMain (3).ipynb 48/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [41]:  test_df.head(10)

Out[41]:

PassengerId Pclass Sex Age Fare Embarked Title IsAlone Age*Class

0 892 3 0 2 0 2 1 1 6

1 893 3 1 2 0 0 3 0 6

2 894 2 0 3 1 2 1 1 6

3 895 3 0 1 1 0 1 1 3

4 896 3 1 1 1 0 3 0 3

5 897 3 0 0 1 0 1 1 0

6 898 3 1 1 0 2 2 1 3

7 899 2 0 1 2 0 1 0 2

8 900 3 1 1 0 1 3 1 3

9 901 3 0 1 2 0 1 0 3
 In [80]:  newdf.head(10)

Out[80]:

Age Embarked Fare Pclass Sex Survived Title IsAlone Age*Class

0 1 0 0 3 0 0.0 1 0 3

1 2 1 3 1 1 1.0 3 0 2

2 1 0 1 3 1 1.0 2 1 3

3 2 0 3 1 1 1.0 3 0 2

4 2 0 1 3 0 0.0 1 1 6

5 1 2 1 3 0 0.0 1 1 3

6 3 0 3 1 0 0.0 1 1 3

7 0 0 2 3 0 0.0 4 0 0

8 1 0 1 3 1 1.0 3 0 3

9 0 1 2 2 1 1.0 3 0 0

Model, predict and solve

Now we are ready to train a model and predict the required solution. There are 60+ predictive modelling
algorithms to choose from. We must understand the type of problem and solution requirement to narrow down
to a select few models which we can evaluate. Our problem is a classification and regression problem. We want
to identify relationship between output (Survived or not) with other variables or features (Gender, Age, Port...).
We are also perfoming a category of machine learning which is called supervised learning as we are training
our model with a given dataset. With these two criteria - Supervised Learning plus Classification and
Regression, we can narrow down our choice of models to a few. These include:

localhost:8888/notebooks/titanicMain (3).ipynb 49/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

Logistic Regression
KNN or k-Nearest Neighbors
Support Vector Machines
Naive Bayes classifier
Decision Tree
Random Forrest
Perceptron
Artificial neural network
RVM or Relevance Vector Machine

In [42]: 

X_train = train_df.drop("Survived", axis=1)

Y_train = train_df["Survived"]
X_test = test_df.drop("PassengerId", axis=1).copy()
X_train.shape, Y_train.shape, X_test.shape

Out[42]:

((891, 8), (891,), (418, 8))

In [82]: 

X_newdf = newdf.drop("Survived", axis=1)

 Y_newdf = newdf["Survived"]

X_newdf.shape, Y_newdf.shape

Out[82]:

((1309, 8), (1309,))

In [43]: 

# create training and testing vars

from sklearn.model_selection import train_test_split

Features = X_train
Class = Y_train

Feature_Train, Feature_Test, Class_Train, Class_Test = train_test_split(X_train, Y_train,

print (Feature_Train.shape, Class_Train.shape)
print (Feature_Test.shape, Class_Test.shape)

# X_train_train, X_test_train, Y_train_train, Y_test_train = train_test_split(X_train, Y_tr

# print X_train_train.shape, Y_train_train.shape
# print X_test_train.shape, Y_test_train.shape

(712, 8) (712,)
(179, 8) (179,)

localhost:8888/notebooks/titanicMain (3).ipynb 50/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [151]: 

# create training and testing vars

from sklearn.model_selection import train_test_split

Features = X_newdf
Class = Y_newdf

Feature_Train, Feature_Test, Class_Train, Class_Test = train_test_split(X_newdf, Y_newdf,

print (Feature_Train.shape, Class_Train.shape)
print (Feature_Test.shape, Class_Test.shape)

# X_train_train, X_test_train, Y_train_train, Y_test_train = train_test_split(X_newdf, Y_ne

# print X_train_train.shape, Y_train_train.shape
# print X_test_train.shape, Y_test_train.shape

(1047, 8) (1047,)
(262, 8) (262,)

Logistic Regression is a useful model to run early in the workflow. Logistic regression measures the relationship
between the categorical dependent variable (feature) and one or more independent variables (features) by
estimating probabilities using a logistic function, which is the cumulative logistic distribution. Reference
Wikipedia (https://en.wikipedia.org/wiki/Logistic_regression).
 Note the confidence score generated by the model based on our training dataset.

In [44]:  # Logistic Regression without Split

logreg = LogisticRegression()
logreg.fit(X_train, Y_train)
Y_pred = logreg.predict(X_test)
acc_log = round(logreg.score(X_train, Y_train) * 100, 2)
acc_log

Out[44]:

80.359999999999999

In [45]:  pd.crosstab(logreg.predict(X_train),Y_train)

Out[45]:

Survived 0 1

row_0

0 479 105

1 70 237

Explanation for the above:

localhost:8888/notebooks/titanicMain (3).ipynb 51/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

0-to-0 (479) is correctly predicted (not survived)

0-to-1 (105) is predicted (not survived) but actually it is survived

1-to-1 (237) is the correct prediction of survival (survived)

1-to-0 (70) predicted to survived but ideally it is not survived

In [46]:  # Logistic Regression with Split

logreg = LogisticRegression()
logreg.fit(Feature_Train, Class_Train)
Y_pred = logreg.predict(Feature_Test)
acc_log = round(logreg.score(Feature_Test, Class_Test) * 100, 2)
acc_log

Out[46]:

81.010000000000005

In [ ]: 

from tpot import TPOTClassifier

from sklearn.model_selection import train_test_split

#X_train, X_test, y_train, y_test = train_test_split(X_train, Y_train,train_size=0.75, test

tpot = TPOTClassifier(generations=5, population_size=20, verbosity=2)

tpot.fit(X_train, y_train)
 print(tpot.score(X_test, y_test))
tpot.export('tpot_mnist_pipeline.py')

We can use Logistic Regression to validate our assumptions and decisions for feature creating and completing
goals. This can be done by calculating the coefficient of the features in the decision function.

Positive coefficients increase the log-odds of the response (and thus increase the probability), and negative
coefficients decrease the log-odds of the response (and thus decrease the probability).

Sex is highest positivie coefficient, implying as the Sex value increases (male: 0 to female: 1), the
probability of Survived=1 increases the most.
Inversely as Pclass increases, probability of Survived=1 decreases the most.
This way Age*Class is a good artificial feature to model as it has second highest negative correlation with
Survived.
So is Title as second highest positive correlation.

localhost:8888/notebooks/titanicMain (3).ipynb 52/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [47]: 

coeff_df = pd.DataFrame(train_df.columns.delete(0))
coeff_df.columns = ['Feature']
coeff_df["Correlation"] = pd.Series(logreg.coef_[0])

coeff_df.sort_values(by='Correlation', ascending=False)

Out[47]:

Feature Correlation

1 Sex 2.128733

5 Title 0.394961

4 Embarked 0.310878

2 Age 0.261064

6 IsAlone 0.242516

3 Fare -0.000617

7 Age*Class -0.277807

0 Pclass -0.733955

Next we model using Support Vector Machines which are supervised learning models with associated learning
algorithms that analyze data used for classification and regression analysis. Given a set of training samples,
each marked as belonging to one or the other of two categories, an SVM training algorithm builds a model that
assigns new test samples to one category or the other, making it a non-probabilistic binary linear classifier.
Reference Wikipedia (https://en.wikipedia.org/wiki/Support_vector_machine).

Note that the model generates a confidence score which is higher than Logistics Regression model.
 In [48]:  # Support Vector Machines without Split

svc = SVC()
svc.fit(X_train, Y_train)
Y_pred = svc.predict(X_test)
acc_svc = round(svc.score(X_train, Y_train) * 100, 2)
acc_svc

Out[48]:

83.840000000000003

localhost:8888/notebooks/titanicMain (3).ipynb 53/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [49]:  # Support Vector Machines with Split

svc = SVC()
svc.fit(Feature_Train, Class_Train)
Y_pred = svc.predict(Feature_Test)
acc_svc = round(svc.score(Feature_Test, Class_Test) * 100, 2)
acc_svc

Out[49]:

82.680000000000007

In pattern recognition, the k-Nearest Neighbors algorithm (or k-NN for short) is a non-parametric method used
for classification and regression. A sample is classified by a majority vote of its neighbors, with the sample
being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically
small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor. Reference
Wikipedia (https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm).

KNN confidence score is better than Logistics Regression but worse than SVM.

In [50]: 

knn = KNeighborsClassifier(n_neighbors = 3)
knn.fit(X_train, Y_train)
Y_pred = knn.predict(X_test)
acc_knn = round(knn.score(X_train, Y_train) * 100, 2)
acc_knn

Out[50]:

84.739999999999995

In machine learning, naive Bayes classifiers are a family of simple probabilistic classifiers based on applying
 Bayes' theorem with strong (naive) independence assumptions between the features. Naive Bayes classifiers
are highly scalable, requiring a number of parameters linear in the number of variables (features) in a learning
problem. Reference Wikipedia (https://en.wikipedia.org/wiki/Naive_Bayes_classifier).

The model generated confidence score is the lowest among the models evaluated so far.

In [51]:  # Gaussian Naive Bayes

gaussian = GaussianNB()
gaussian.fit(X_train, Y_train)
Y_pred = gaussian.predict(X_test)
acc_gaussian = round(gaussian.score(X_train, Y_train) * 100, 2)
acc_gaussian

Out[51]:

72.280000000000001

The perceptron is an algorithm for supervised learning of binary classifiers (functions that can decide whether
localhost:8888/notebooks/titanicMain (3).ipynb 54/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

an input, represented by a vector of numbers, belongs to some specific class or not). It is a type of linear
classifier, i.e. a classification algorithm that makes its predictions based on a linear predictor function combining
a set of weights with the feature vector. The algorithm allows for online learning, in that it processes elements in
the training set one at a time. Reference Wikipedia (https://en.wikipedia.org/wiki/Perceptron).

In [52]:  # Perceptron

perceptron = Perceptron()
perceptron.fit(X_train, Y_train)
Y_pred = perceptron.predict(X_test)
acc_perceptron = round(perceptron.score(X_train, Y_train) * 100, 2)
acc_perceptron

C:\Users\BigDataLab\Anaconda2\envs\mudi\lib\site-packages\sklearn\linear_mod
el\stochastic_gradient.py:84: FutureWarning: max_iter and tol parameters hav
e been added in <class 'sklearn.linear_model.perceptron.Perceptron'> in 0.1
9. If both are left unset, they default to max_iter=5 and tol=None. If tol i
s not None, max_iter defaults to max_iter=1000. From 0.21, default max_iter
will be 1000, and default tol will be 1e-3.
"and default tol will be 1e-3." % type(self), FutureWarning)

Out[52]:

78.0

In [53]:  # Linear SVC

linear_svc = LinearSVC()
linear_svc.fit(X_train, Y_train)
Y_pred = linear_svc.predict(X_test)
acc_linear_svc = round(linear_svc.score(X_train, Y_train) * 100, 2)
acc_linear_svc

Out[53]:

79.120000000000005
 localhost:8888/notebooks/titanicMain (3).ipynb 55/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [54]:  # Stochastic Gradient Descent

sgd = SGDClassifier()
sgd.fit(X_train, Y_train)
Y_pred = sgd.predict(X_test)
acc_sgd = round(sgd.score(X_train, Y_train) * 100, 2)
acc_sgd

C:\Users\BigDataLab\Anaconda2\envs\mudi\lib\site-packages\sklearn\linear_mod
el\stochastic_gradient.py:84: FutureWarning: max_iter and tol parameters hav
e been added in <class 'sklearn.linear_model.stochastic_gradient.SGDClassifi
er'> in 0.19. If both are left unset, they default to max_iter=5 and tol=Non
e. If tol is not None, max_iter defaults to max_iter=1000. From 0.21, defaul
t max_iter will be 1000, and default tol will be 1e-3.
"and default tol will be 1e-3." % type(self), FutureWarning)

Out[54]:

74.069999999999993

This model uses a decision tree as a predictive model which maps features (tree branches) to conclusions
about the target value (tree leaves). Tree models where the target variable can take a finite set of values are
called classification trees; in these tree structures, leaves represent class labels and branches represent
conjunctions of features that lead to those class labels. Decision trees where the target variable can take
continuous values (typically real numbers) are called regression trees. Reference Wikipedia
(https://en.wikipedia.org/wiki/Decision_tree_learning).

The model confidence score is the highest among models evaluated so far.

In [55]:  # Decision Tree

decision_tree = DecisionTreeClassifier()
decision_tree.fit(X_train, Y_train)
Y_pred = decision_tree.predict(X_test)
acc_decision_tree = round(decision_tree.score(X_train, Y_train) * 100, 2)
acc_decision_tree

Out[55]:

86.760000000000005

localhost:8888/notebooks/titanicMain (3).ipynb 56/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [56]: 
 # print Feature_Train.shape, Class_Train.shape
# print Feature_Test.shape, Class_Test.shape

# Decision Tree

decision_tree = DecisionTreeClassifier()
decision_tree.fit(Feature_Train, Class_Train)
Y_pred = decision_tree.predict(X_test)
acc_decision_tree = round(decision_tree.score(Feature_Test, Class_Test) * 100, 2)
acc_decision_tree

Out[56]:

81.560000000000002

In [59]:  print(pd.crosstab(decision_tree.predict(Feature_Test), Class_Test))

Survived 0 1
row_0
0 95 23
1 10 51

In [60]: 

prediction =decision_tree.predict(Feature_Test)
type=(Class_Test)
y=pd.DataFrame(Class_Test)
y['PredictedByModel']=prediction
combine =[Feature_Test,y]
newdf= pd.concat(combine, axis=1)
print (newdf.head())
newdf.to_csv('titanic_Output.csv')

Pclass Sex Age Fare Embarked Title IsAlone Age*Class Survived \

783 3 0 1 2 0 1 0 3 0 347 3 1 1 2 0 3 0 3 1 623 3 0 1 0 0 1 1 3 0 246 3 1
1 0 0 2 1 3 0 309 1 1 1 3 1 2 1 1 1

PredictedByModel
783 0
347 0
623 0
246 1
309 1

The next model Random Forests is one of the most popular. Random forests or random decision forests are an
ensemble learning method for classification, regression and other tasks, that operate by constructing a
multitude of decision trees (n_estimators=100) at training time and outputting the class that is the mode of the

localhost:8888/notebooks/titanicMain (3).ipynb 57/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

classes (classification) or mean prediction (regression) of the individual trees. Reference Wikipedia
(https://en.wikipedia.org/wiki/Random_forest).

The model confidence score is the highest among models evaluated so far. We decide to use this model's
output (Y_pred) for creating our competition submission of results.

In [63]:  # Random Forest

 random_forest = RandomForestClassifier(n_estimators=100)
random_forest.fit(X_train, Y_train)
Y_pred = random_forest.predict(X_test)
random_forest.score(X_train, Y_train)
acc_random_forest = round(random_forest.score(X_train, Y_train) * 100, 2)
print (acc_random_forest)

86.76

In [65]:  print(pd.crosstab(random_forest.predict(X_train), Y_train))

Survived 0 1
row_0
0 500 69
1 49 273

In [66]:  # Random Forest

random_forest = RandomForestClassifier(n_estimators=100)
random_forest.fit(Feature_Train, Class_Train)
Y_pred = random_forest.predict(X_test)
random_forest.score(Feature_Test, Class_Test)
acc_random_forest = round(random_forest.score(Feature_Test, Class_Test) * 100, 2)
print (acc_random_forest)

81.01

In [67]:  print(pd.crosstab(random_forest.predict(Feature_Test), Class_Test))

Survived 0 1
row_0
0 92 21
1 13 53

Model evaluation
We can now rank our evaluation of all the models to choose the best one for our problem. While both Decision
Tree and Random Forest score the same, we choose to use Random Forest as they correct for decision trees'
habit of overfitting to their training set.
localhost:8888/notebooks/titanicMain (3).ipynb 58/60
10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [68]: 

# Logistic Regression with CrossValidation

from sklearn import cross_validation
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import ShuffleSplit

logreg = LogisticRegression(random_state=4)
cv = ShuffleSplit(n_splits=5, test_size=0.2, random_state=50)

scores = cross_val_score(logreg, Feature_Train, Class_Train, scoring='f1', cv=cv)

print (scores)
print (scores.mean())
 [ 0.77477477 0.78181818 0.74226804 0.64864865 0.74137931]
0.737777791365

C:\Users\BigDataLab\Anaconda2\envs\mudi\lib\site-packages\sklearn\cross_vali
dation.py:41: DeprecationWarning: This module was deprecated in version 0.18
in favor of the model_selection module into which all the refactored classes
and functions are moved. Also note that the interface of the new CV iterator
s are different from that of this module. This module will be removed in 0.2
0.
"This module will be removed in 0.20.", DeprecationWarning)

In [70]: 

# Logistic Regression with CrossValidation

from sklearn import cross_validation
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import KFold
from sklearn.model_selection import cross_val_predict

import numpy as np
print (Feature_Train.shape[0])
rf = RandomForestClassifier(random_state=1, n_estimators=100)
kf = KFold(Feature_Train.shape[0], n_folds=5, random_state=1)
#cv = ShuffleSplit(n_splits=5, test_size=0.2, random_state=50)

predictions = cross_validation.cross_val_predict(rf, Feature_Train, Class_Train, cv=kf)

predictions = pd.Series(predictions)
scores = cross_val_score(rf, Feature_Train, Class_Train, scoring='f1', cv=kf)

# predictions_test = cross_validation.cross_val_predict(rf,Feature_Test,Class_Test, cv=kf)

# Take the mean of the scores (because we have one for each fold)
print (scores)
print (scores.mean())

712
[ 0.69565217 0.73076923 0.80373832 0.69811321 0.75 ]
0.735654585997

localhost:8888/notebooks/titanicMain (3).ipynb 59/60

10/26/21, 5:41 PM titanicMain (3) - Jupyter Notebook

In [ ]:  from sklearn.metrics import precision_score

2
Type Markdown and LaTeX: ��

2
Type Markdown and LaTeX: ��
localhost:8888/notebooks/titanicMain (3).ipynb 60/60