Lab 08
Data Preprocessing
Objective:
The objective of this lab is to learn how to Preprocess data before applying any machine learning tool.
Activity Outcomes:
On completion of this lab student will be able to
Fill in Missing values
Deal with Categorical Data
Perform Normalization of Dataset for improved results
Split the Dataset
Do feature Scaling
Instructor Note:
As pre-lab activity, read chapter 2 from the text book “Learning Data Mining with Python, By Robert
Layton, PACKT Publishing”.
1) Useful Concepts
Data Preprocessing
In a real-world data science project, data preprocessing is one of the most important things, and it is one
of the common factors of success of a model, i.e., if there is correct data preprocessing and feature
engineering, that model is more likely to produce noticeably better results as compared to a model for
which data is not well preprocessed.
There are 4 main important steps for the preprocessing of data.
Taking care of Missing values
Taking care of Categorical Features
Normalization of data set
Splitting of the data set in Training and Validation sets
1. Taking Care of Missing Values
There is a famous Machine Learning phrase which you might have heard that is
Garbage in Garbage out
If your data set is full of NaNs and garbage values, then surely your model will perform garbage too. So
taking care of such missing values is important.
You can add following library for handling missing values
from sklearn.impute import SimpleImputer
2. Taking care of Categorical Features
We can take care of categorical features by converting them to integers. There are 2 common ways to do
so.
1. Label Encoding
2. One Hot Encoding
In Label Encoder, we can convert the Categorical values into numerical labels. In OneHotEncoder we
make a new column for each unique categorical value, and the value is 1 for that column, if in an actual
data frame that value is there, else it is 0.
You will use following library for taking care of categorical features
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
3. Normalizing the Dataset
This brings us to the last part of data preprocessing, which is the normalization of the dataset. It is proven
from certain experimentation that Machine Learning and Deep Learning Models perform way better on a
normalized data set as compared to a data set that is not normalized.
The goal of normalization is to change values to a common scale without distorting the difference
between the range of values.
There are several ways to do so. We will discuss 2 common ways to normalize a dataset.
Standard Scaler
Standardization :
x−μ
z=
σ
With mean :
N
1
μ= ∑ (x )
N i=1 i
¿ standard deviation
√
N
1
σ= ∑ ( x −μ )2
N i=1 i
You will use following library for normalizing datasets.
from sklearn.preprocessing import StandardScaler
4. Train Test Split
Train Test Split is one of the important steps in Machine Learning. It is very important because your
model needs to be evaluated before it has been deployed. And that evaluation needs to be done on unseen
data because when it is deployed, all incoming data is unseen.
The main idea behind the train test split is to convert original data set into 2 parts
train
test
where train consists of training data and training labels and test consists of testing data and testing labels.
Following command is used for splitting training and testing data
from sklearn.model_selection import train_test_split
2) Solved Lab Activities
Sr. No Allocated Time Level of Complexity CLO Mapping
Activity 1 10 minutes Low CLO-5
Activity 2 10 minutes Medium CLO-5
Activity 3 10 minutes Medium CLO-5
Activity 4 10 minutes Medium CLO-5
Activity 5 10 minutes Medium CLO-5
Activity 1: Importing the libraries and Dataset
# libraries
import numpy as np
# used for handling numbers
import pandas as pd
# used for handling the dataset
from sklearn.impute import SimpleImputer
# used for handling missing data
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
# used for encoding categorical data
from sklearn.model_selection import train_test_split
# used for splitting training and testing data
from sklearn.preprocessing import StandardScaler
# used for feature scaling
If you see any import errors, try to install those packages explicitly using pip command as follows.
pip install <package-name>
Importing the Dataset
First of all, let us have a look at the dataset we are going to use for this particular example. You can find
the dataset here.
Figure 8.1: Sample dataset
In order to import this dataset into our script, we are apparently going to use pandas as follows.
dataset = pd.read_csv('Data.csv')
# to import the dataset into a variable
# Splitting the attributes into independent and dependent attributes
X = dataset.iloc[:, :-1].values
# attributes to determine dependent variable / Class
Y = dataset.iloc[:, -1].values
# dependent variable / Class
When you run this code section, you should not see any errors, if you do make sure the script and
the Data.csv are in the same folder. When successfully executed, you can move to variable explorer in the
Spyder UI and you will see the following three variables.
Figure 8.2. Description of varaiables created after running the code
When you double click on each of these variables, you should see something similar.
Figure 8.3: Detailed description of dataset loaded
Figure 8.4: Detailed description of variable x and y
Activity 2: Handling of Missing Data
Well the first idea is to remove the lines in the observations where there is some missing data. But that can
be quite dangerous because imagine this data set contains crucial information. It would be quite dangerous
to remove an observation. So we need to figure out a better idea to handle this problem. And another idea
that’s actually the most common idea to handle missing data is to take the mean of the columns.
If you noticed in our dataset, we have two values missing, one for age column in 7th data row and for
Income column in 5th data row. Missing values should be handled during the data analysis. So, we do that
as follows.
# handling the missing data and replace missing values with nan from
numpy and replace with mean of all the other values
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
imputer = imputer.fit(X[:, 1:])
X[:, 1:] = imputer.transform(X[:, 1:])
After execution of this code, the independent variable X will transform into the following.
Figure 8.5: Description of independent variable X
Here you can see, that the missing values have been replaced by the average values of the respective
columns.
Activity 3: Handling of Categorical Data
In this dataset we can see that we have two categorical variables. We have the Region variable and the
Online Shopper variable. These two variables are categorical variables because simply they contain
categories. The Region contains three categories. It’s India, USA & Brazil and the online shopper variable
contains two categories. Yes and No that’s why they’re called categorical variables.
You can guess that since machine learning models are based on mathematical equations you can intuitively
understand that it would cause some problem if we keep the text here in the categorical variables in the
equations because we would only want numbers in the equations. So that’s why we need to encode the
categorical variables. That is to encode the text that we have here into numbers. To do this we use the
following code snippet.
# encode categorical data
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
onehotencoder = OneHotEncoder(categorical_features=[0])
X = onehotencoder.fit_transform(X).toarray()
labelencoder_Y = LabelEncoder()
Y = labelencoder_Y.fit_transform(Y)
After execution of this code, the independent variable X and dependent variable Y will transform into the
following.
Figure 8.6: Description of independent variable X and dependent variable Y
Here, you can see that the Region variable is now made up of a 3 bit binary variable. The left most bit
represents India, 2nd bit represents Brazil and the last bit represents USA. If the bit is 1 then it represents
data for that country otherwise not. For Online Shopper variable, 1 represents Yes and 0 represents No.
Activity 4: Splitting the dataset into training and testing datasets
Any machine learning algorithm needs to be tested for accuracy. In order to do that, we divide our data set
into two parts: training set and testing set. As the name itself suggests, we use the training set to make the
algorithm learn the behaviours present in the data and check the correctness of the algorithm by testing on
testing set. In Python, we do that as follows:
# splitting the dataset into training set and test set
X_train, X_test, Y_train, Y_test = train_test_split(X, Y,
test_size=0.2, random_state=0)
Here, we are taking training set to be 80% of the original data set and testing set to be 20% of the original
data set. This is usually the ratio in which they are split. But, you can come across sometimes to a 70–30%
or 75–25% ratio split. But, you don’t want to split it 50–50%. This can lead to Model Overfitting.
Activity 5: Feature Scaling
As you can see we have these two columns age and income that contains numerical numbers. You notice
that the variables are not on the same scale because the age are going from 32 to 55 and the salaries going
from 57.6 K to like 99.6 K.
So because this age variable in the salary variable don’t have the same scale. This will cause some issues in
your machinery models. And why is that. It’s because your machine models a lot of machinery models are
based on what is called the Euclidean distance.
We use feature scaling to convert different scales to a standard scale to make it easier for Machine
Learning algorithms. We do this in Python as follows:
# feature scaling
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)
3) Graded Lab Tasks
Note: The instructor can design graded lab activities according to the level of difficult and complexity
of the solved lab activities. The lab tasks assigned by the instructor should be evaluated in the same
lab.
1. Download any data set from following links
a. Office Suply Sales sample data workbook
b. get the hockey player data file
c. get the food sales data file
and apply preprocessing steps on that data to make the data ready for applying machine learning
techniques.
