Machine Learning
Lesson 2: Data Wrangling and Manipulation
© Simplilearn. All rights reserved.
Concepts Covered
Data acquisition
Data exploration techniques
Data wrangling techniques
Data manipulation techniques
Typecasting
Learning Objectives
By the end of this lesson, you will be able to:
Demonstrate data import and exploration using Python
Demonstrate different data wrangling techniques and their significance
Perform data manipulation in python using coercion, merging, concatenation,
and joins
Data Preprocessing
Topic 1: Data Exploration
Loading .csv File in Python
Before starting with a dataset, the first step is to load the dataset. Below is the code for the
same:
CSV File Program Data Program
Code
df = pandas.read_csv("/home/simpy/Datasets/BostonHousing.csv")
Path to file
Loading Data to .csv File
Below is the code for loading the data within an existing csv file:
Program Program Data CSV File
Code
df.to_csv("/home/simpy/Datasets/BostonHousing.csv")
Path to file
Loading .xlsx File in Python
Below is the code for loading an xlsx file within python:
XLS File Program Data Program
Code
df = pandas.read_excel("/home/simpy/Datasets/BostonHousing.xlsx")
Loading Data to .xlsx File
Below is the code for loading program data into an existing xlsx file:
Program Program Data XLS File
Code
df.to_excel("/home/simpy/Datasets/BostonHousing.xlsx")
Assisted Practice
Data Exploration Duration: 5 mins.
Problem Statement: Extract data from the given SalaryGender CSV file and store the data from
each column in a separate NumPy array.
Objective: Import the dataset (csv) in/from your Python notebook to local system.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and
password that are generated. Click on the Launch Lab button. On the page that appears, enter the
username and password in the respective fields, and click Login.
Data Exploration Techniques
Dimensionality Check The shape attribute returns a two-item tuple (number of rows and the
number of columns) for the data frame. For a Series, it returns a one-item
Type of Dataset tuple.
Code
Slicing and Indexing
df.shape
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode
Data Exploration Techniques (Contd.)
Dimensionality Check
You can use the type ( ) in python to return the type of object.
Type of Dataset
Checking the type of data frame:
Slicing and Indexing
Code
Identifying Unique type(df)
Elements
Value Extraction
Checking the type of a column (çhas) within a data frame:
Feature Mean
Code
Feature Median df['chas'].dtype
Feature Mode
Data Exploration Techniques (Contd.)
Dimensionality Check
You can use the : operator with the start index on left and end index on
right of it to output the corresponding slice.
Type of Dataset
Slicing a list: list = [1,2,3,4,5]
Slicing and Indexing Code
Identifying Unique list[1:3]
Elements
Value Extraction Slicing a Data frame (df) using iloc indexer:
Feature Mean Code
df.iloc[:,1:3]
Feature Median
Feature Mode
Data Exploration Techniques (Contd.)
Dimensionality Check
Using unique ( ) on the column of interest will return a numpy array with
unique values of the column.
Type of Dataset
Extracting all unique values out of ‘’crim” column:
Slicing and Indexing Code
Identifying Unique df['crim'].unique()
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode
Data Exploration Techniques (Contd.)
Dimensionality Check
Using value ( ) on the column of interest will return a numpy array with all
the values of the column.
Type of Dataset
Extracting values out of ‘’crim” column:
Slicing and Indexing Code
Identifying Unique df['crim’].values()
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode
Data Exploration Techniques (Contd.)
Dimensionality Check
Using mean( ) on the data frame will return mean of the data frame across
all the columns.
Type of Dataset
Code
Slicing and Indexing df.mean()
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode
Data Exploration Techniques (Contd.)
Dimensionality Check
Using median( ) on the data frame will return median values of the data
frame across all the columns.
Type of Dataset
Code
Slicing and Indexing df.median()
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode
Data Exploration Techniques (Contd.)
Dimensionality Check
Using mode( ) on the data frame will return mode values of the data frame
across all the columns, rows with axis=0 and axis = 1, respectively.
Type of Dataset
Code
Slicing and Indexing df.mode(axis=0)
Identifying Unique
Elements
Value Extraction
Feature Mean
Feature Median
Feature Mode
Let’s now consider multiple features and understand the effect of one over other with
respect to correlation (using seaborn)
©Simplilearn. All rights reserved
Seaborn is a library for making
attractive and informative statistical
graphics in Python. It is built on top of
matplotlib and integrated with the
PyData Stack, including support for
numpy and pandas data structures, and
statistical routines.
©Simplilearn. All rights reserved
Plotting a Heatmap with Seaborn
Below is the code for plotting a heatmap within Python:
Code
import matplotlib.pyplot as plt
import seaborn as sns
correlations = df.corr()
sns.heatmap(data = correlations,square = True, cmap = "bwr")
plt.yticks(rotation=0)
plt.xticks(rotation=90)
Rectangular dataset (2D
dataset that can be Matplotlib
If True, set the Axes
coerced into an ndarray) colormap name or
aspect to “equal” so
object, or list of
each cell will be
colors
square-shaped
Plotting a Heatmap with Seaborn (Contd.)
Below is the heatmap obtained, where, approaching red colour means maximum correlation
and approaching blue means minimal correlation.
Minimum correlation
Maximum correlation
Assisted Practice
Data Exploration Duration: 15 mins.
Problem Statement: Suppose you are a public school administrator. Some schools in your state of Tennessee
are performing below average academically. Your superintendent under pressure from frustrated parents and
voters approached you with the task of understanding why these schools are under-performing. To improve
school performance, you need to learn more about these schools and their students, just as a business needs to
understand its own strengths and weaknesses and its customers. The data includes various demographic, school
faculty, and income variables.
Objective: Perform exploratory data analysis which includes: determining the type of the data, correlation
analysis over the same. You need to convert the data into useful information:
▪ Read the data in pandas data frame
▪ Describe the data to find more details
▪ Find the correlation between ‘reduced_lunch’ and ‘school_rating’
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that
are generated. Click on the Launch Lab button. On the page that appears, enter the username and password in
the respective fields, and click Login.
Unassisted Practice
Data Exploration Duration: 15
mins.
Problem Statement: Mtcars, an automobile company in Chambersburg, United States has recorded the
production of its cars within a dataset. With respect to some of the feedback given by their customers they are
coming up with a new model. As a result of it they have to explore the current dataset to derive further insights out if
it.
Objective: Import the dataset, explore for dimensionality, type and average value of the horsepower across all the
cars. Also, identify few of mostly correlated features which would help in modification.
Note: This practice is not graded. It is only intended for you to apply the knowledge you have gained to solve real-
world problems.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login
Data Import
The first step is to import the data as a part of exploration.
Code
df1 = pandas.read_csv(“mtcars.csv“)
Data Exploration
The shape property is usually used to get the current shape of an
array/df.
Dimensionality Check Code
df1.shape
Type of Dataset
Identifying mean value
Data Exploration
type(), returns type of the given object.
Dimensionality Check Code
type(df1)
Type of Dataset
Identifying mean value
Data Exploration
mean( ) function can be used to calculate mean/average of a given list of numbers.
Dimensionality Check Code
df1[‘hp’].mean()
Type of Dataset
Identifying mean value
Identifying Correlation Using a Heatmap
Heatmap function in seaborn is used to plot the correlation matrix.
Code
import matplotlib.pyplot as plt
import seaborn as sns
correlations = df1.corr()
sns.heatmap(data = correlations,square = True, cmap = “viridis")
plt.yticks(rotation=0)
plt.xticks(rotation=90)
Identifying Correlation Using a Heatmap
Graphical representation of data where the individual values contained in a
matrix are represented in colors.
From the adjacent map, you
can clearly see that
cylinder (cyl) and
displacement (disp) are the
most correlated features.
Data Preprocessing
Topic 2: Data Wrangling
Data Wrangling
The process of manually converting or mapping data from one raw format into another format is called data wrangling. This
includes munging and data visualization.
Discovering Structuring
Different Tasks
in Data Cleaning
Wrangling
Enrichment
Validating
Need of Data Wrangling
Following are the problems that can be avoided with wrangled data:
Missing data, a very common problem
Presence of noisy data (erroneous data and outliers)
Inconsistent data
Develop a more accurate model
Prevent data leakage
Missing Values in a Dataset
Consider a random dataset given below, illustrating
missing values.
Missing Value Detection
Consider a dataset below, imported as df1 within Python, having some missing values.
Code
Detecting
missing df1.isna().any()
values
Missing Value Treatment
Code
Mean Imputation: Replace the missing value from sklearn.preprocessing import Imputer
with variable’s mean mean_imputer =
Imputer(missing_values=np.nan,strategy='mean',axis=1)
mean_imputer = mean_imputer.fit(df1)
imputed_df = mean_imputer.transform(df1.values)
df1 = pd.DataFrame(data=imputed_df,columns=cols)
df1
Missing Value Treatment (Contd.)
Code
Mean Imputation: Replace the missing value from sklearn.preprocessing import Imputer
with variable’s mean median_imputer=Imputer(missing_values=np.nan,strategy
=‘median',axis=1)
median_imputer = median_imputer.fit(df1)
imputed_df = median_imputer.transform(df1.values)
df1 = pd.DataFrame(data=imputed_df,columns=cols)
Median Imputation: Replace the missing df1
value with variable’s median
Note: Mean imputation/Median imputation is again model dependent and is valid only on numerical data.
Outlier Values in a Dataset
4
FREQUENCY
3 OUTLIER?
An outlier is a value that lies
2
outside the usual observation of
1 values.
0
7.0 7.5 8.0 8.5 9.0 9.5 10.0 10.5 11.0 11.5 12.0 12.5 13.0 13.5 14.0 14.5 15.0
x MIDPOINT
13
12
11
10
Y3
9
8
7
6
5
5.0 7.5 10.0 12.5 15.0
X1
Note: Outliers skew the data when you are trying to do any type of average.
Dealing with an Outlier
Detect any outlier in the first column of df1
Outlier Detection
Code
import seaborn as sns
sns.boxplot(x=df1['Assignment'])
Outlier Treatment
Outliers:
Values < 60
Dealing with an Outlier
Create a filter based on the boxplot obtained
and apply the filter to the data frame
Outlier Detection
Code
filter=df1['Assignment'].values>60
Outlier Treatment df1_outlier_rem=df1[filter]
df1_outlier_rem
Assisted Practice
Data Wrangling Duration: 15 mins.
Problem Statement: Load the load_diabetes datasets internally from sklearn and check for any missing value or
outlier data in the ‘data’ column. If any irregularities found treat them accordingly.
Objective: Perform missing value and outlier data treatment.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
Unassisted Practice
Data Wrangling Duration: 5 mins.
Problem Statement: Mtcars, the automobile company in the United States have planned to rework on optimizing
the horsepower of their cars, as most of the customers feedbacks were centred around horsepower. However, while
developing a ML model with respect to horsepower, the efficiency of the model was compromised. Irregularity might
be one of the causes.
Objective: Check for missing values and outliers within the horsepower column and remove them.
Note: This practice is not graded. It is only intended for you to apply the knowledge you have gained to solve real-
world problems.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
Check for Irregularities
Check for missing values Check for Outliers
Code Code
df1['hp'].isna().any() sns.boxplot(x=df1['hp'])
Outlier
Outlier Treatment
Data with hp>250 is the outlier data. Therefore, you can filter it accordingly.
Code
filter = df1['hp']<250
df1_out_rem = df1[filter]
sns.boxplot(x=df2_out_rem['hp'])
Outlier filtered
data
Data Preprocessing
Topic 3: Data Manipulation
Functionalities of Data Object in Python
A data object is a two-dimensional data structure, i.e., data is aligned in a tabular fashion in rows and
columns.
head( )
tail( )
values( )
groupby( )
Concatenation
Merging
Functionalities of Data Object in Python (Contd.)
head( ) Head( ) returns the first n rows of the data structure
tail( ) Code
values( ) import pandas as pd
import numpy as np
df=pd.Series(np.arange(1,51))
print(df.head(6))
groupby( )
Concatenation
Merging
Functionalities of Data Object in Python (Contd.)
head( ) Tail( ) returns the last n rows of the data structure
tail( ) Code
values( ) import pandas as pd
import numpy as np
groupby( ) df=pd.Series(np.arange(1,51))
print(df.tail(6))
Concatenation
Merging
Functionalities of Data Object in Python (Contd.)
head( ) values( ) returns the actual data in the series of the array
tail( ) Code
values( ) import pandas as pd
import numpy as np
groupby( ) df=pd.Series(np.arange(1,51))
print(df.values)
Concatenation
Merging
Functionalities of Data Object using Python (Contd.)
head( ) The Data Frame is grouped according to the ‘Team’ and ‘ICC_Rank’ columns
Code
tail( )
import pandas as pd
values( ) world_cup={'Team':['West Indies','West
indies','India','Australia','Pakistan','Sri
Lanka','Australia','Australia','Australia','
Insia','Australia'],
groupby( )
'Rank':[7,7,2,1,6,4,1,1,1,2,1],
'Year':[1975,1979,1983,1987,1992,1996,1999,2003
,2007,2011,2015]}
Concatenation df=pd.DataFrame(world_cup)
print(df.groupby(['Team','Rank’]).groups)
Merging
Functionalities of Data Object in Python (Contd.)
Concatenation combines two or more data structures.
head( )
Code
tail( )
import pandas
values( ) world_champions={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka’],
'ICC_rank':[2,3,7,8,4],
'World_champions_Year':[2011,2015,1979,1992,1996],
groupby( ) 'Points':[874,787,753,673,855]}
chokers={'Team':['South Africa','New
Zealand','Zimbabwe'],'ICC_rank':[1,5,9],
Concatenation 'Points':[895,764,656]}
df1=pandas.DataFrame(world_champions)
df2=pandas.DataFrame(chokers)
print(pandas.concat([df1,df2],axis=1))
Merging
Functionalities of Data Object in Python (Contd.)
The concatenated output:
head( )
tail( )
values( )
groupby( )
Concatenation
Merging
Functionalities of Data Object in Python (Contd.)
Merging is the Pandas operation that performs database joins on objects
head( )
Code
tail( )
import pandas
values( ) champion_stats={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka'],
'ICC_rank':[2,3,7,8,4],
'World_champions_Year':[2011,2015,1979,1992,1996],
groupby( ) 'Points':[874,787,753,673,855]}
match_stats={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka'],
Concatenation 'World_cup_played':[11,10,11,9,8],
'ODIs_played':[733,988,712,679,662]}
df1=pandas.DataFrame(champion_stats)
df2=pandas.DataFrame(match_stats)
Merging
print(df1)
print(df2)
print(pandas.merge(df1,df2,on='Team'))
Functionalities of Data Object in Python (Contd.)
head( )
tail( )
values( )
groupby( )
Concatenation
Merging The merged object
contains all the columns
of the data frames
merged
Different Types of Joins
Joins are used to combine records from two or more tables in a database. Below
are the four most commonly used joins:
Left Join Right Join Inner Join Full Outer Join
Left Join
Left Join Code
import pandas
world_champions={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka'],
'ICC_rank':[2,3,7,8,4],
'World_champions_Year':[2011,2015,1979,1992,1996],
'Points':[874,787,753,673,855]}
chokers={'Team':['South Africa','New
Zealand','Zimbabwe'],
'ICC_rank':[1,5,9],'Points':[895,764,656]}
Returns all rows from df1=pandas.DataFrame(world_champions)
the left table, even if df2=pandas.DataFrame(chokers)
there are no matches in print(pandas.merge(df1,df2,on='Team',how='left'))
the right table
Right Join
Right Join Code
import pandas
world_champions={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka'],
'ICC_rank':[2,3,7,8,4],
'World_champions_Year':[2011,2015,1979,1992,1996],
'Points':[874,787,753,673,855]}
chokers={'Team':['South Africa','New
Zealand','Zimbabwe'],'ICC_rank':[1,5,9],'Points':[89
5,764,656]}
Preserves the unmatched df1=pandas.DataFrame(world_champions)
rows from the second df2=pandas.DataFrame(chokers)
(right) table, joining them print(pandas.merge(df1,df2,on='Team',how=‘right'))
with a NULL in the shape
of the first (left) table
Inner Join
Inner Join Code
import pandas
world_champions={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka'],
'ICC_rank':[2,3,7,8,4],
'World_champions_Year':[2011,2015,1979,1992,1996],
'Points':[874,787,753,673,855]}
chokers={'Team':['South Africa','New
Zealand','Zimbabwe'],'ICC_rank':[1,5,9],'Points':[89
5,764,656]}
Selects all rows from df1=pandas.DataFrame(world_champions)
both participating tables df2=pandas.DataFrame(chokers)
if there is a match print(pandas.merge(df1,df2,on='Team',how=‘inner'))
between the columns
Full Outer Join
Code
Full Outer Join
import pandas
world_champions={'Team':['India','Australia','West
Indies','Pakistan','Sri Lanka'],
'ICC_rank':[2,3,7,8,4],
'World_champions_Year':[2011,2015,1979,1992,1996],
'Points':[874,787,753,673,855]}
chokers={'Team':['South Africa',’New
Zealand','Zimbabwe'],'ICC_rank':[1,5,9],'Points':[89
5,764,656]}
df1=pandas.DataFrame(world_champions)
Returns all records when
df2=pandas.DataFrame(chokers)
there is a match in either
print(pandas.merge(df1,df2,on='Team',how=‘outer'))
left (table1) or right
(table2) table records
Typecasting
It converts the data type of an object to the required data
type.
string( ) Int( )
Returns string from any Returns an integer object
numeric object or converts from any number or string.
any number to string
float( )
Returns a floating-point
number from a number or a
string
Typecasting Using Int, float and string( )
Few typecasted data types
Code Code
int(12.32) float(23)
Code
int(12.32)
Code Code
float('21.43
int(‘43’)
')
Assisted Practice
Data Manipulation Duration: 10 mins.
Problem Statement: As a macroeconomic analyst at the Organization for Economic Cooperation and Development
(OECD), your job is to collect relevant data for analysis. It looks like you have three countries in the north_america data
frame and one country in the south_america data frame. As these are in two separate plots, it's hard to compare the
average labor hours between North America and South America. If all the countries were into the same data frame, it
would be much easier to do this comparison.
Objective: Demonstrate concatenation.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
Unassisted Practice
Data Manipulation Duration: 10 mins.
Problem Statement: SFO Public Department - referred to as SFO has captured all the salary data of its employees
from year 2011-2014. Now in 2018 the organization is facing some financial crisis. As a first step HR wants to
rationalize employee cost to save payroll budget. You have to do data manipulation and answer the below questions:
1. How much total salary cost has increased from year 2011 to 2014?
2. Who was the top earning employee across all the years?
Objective: Perform data manipulation and visualization techniques
Note: This practice is not graded. It is only intended for you to apply the knowledge you have gained to solve real-
world problems.
Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
Answer 1
Check the mean salary cost per year and see how it has increased per
year.
Code
salary = pd.read_csv('Salaries.csv')
mean_year =
salary.groupby('Year').mean()['TotalPayBenefits']
print ( mean_year)
Answer 2
Group the total salary with respect to employee name:
Code
top_sal =
salary.groupby('EmployeeName').sum()['TotalPayBenefi
ts']
print((top_sal.sort_values(axis=0)))
Key Takeaways
Now, you are able to:
Demonstrate data import and exploration using Python
Demonstrate different data wrangling techniques and their significance
Perform data manipulation in python using coercion, merging,
concatenation, and joins
Knowledge
Check
©Simplilearn. All rights reserved
Knowledge
Check
Which of the following plots can be used to detect an outlier?
1
a. Boxplot
b. Histogram
c. Scatter plot
d. All of the above
Knowledge
Check
Which of the following plots can be used to detect an outlier?
1
a. Boxplot
b. Histogram
c. Scatter plot
d. All of the above
The correct answer is d . All of the above
All the above plots can be used to detect an outlier.
Knowledge
What is the output of the below Python code?
Check
import numpy as np percentiles = [98, 76.37, 55.55, 69, 88]
2 first_subject = np.array(percentiles) print first_subject.dtype
a. float32
b. float
c. int32
d. float64
Knowledge What is the output of the below Python code?
Check import numpy as np
percentiles = [98, 76.37, 55.55, 69, 88]
2 first_subject = np.array(percentiles)
print first_subject.dtype
a. float32
b. float
c. int32
d. float64
The correct answer is d. float64
Float64’s can represent numbers much more accurately than other floats and has more storage capacity.
Lesson-End Project Duration: 20 mins.
Problem Statement: From the raw data below create a data frame:
'first_name': ['Jason', 'Molly', 'Tina', 'Jake', 'Amy'], 'last_name': ['Miller', 'Jacobson', ".", 'Milner', 'Cooze'],
'age': [42, 52, 36, 24, 73], 'preTestScore': [4, 24, 31, ".", "."],'postTestScore': ["25,000", "94,000", 57, 62, 70]
Objective: Perform data processing on raw data:
▪ Save the data frame into a csv file as project.csv
▪ Read the project.csv and print the data frame
▪ Read the project.csv without column heading
▪ Read the project.csv and make the index columns as 'First Name’ and 'Last Name'
▪ Print the data frame in a Boolean form as True or False. True for Null/ NaN values and false for
non-null values
▪ Read the data frame by skipping first 3 rows and print the data frame
Access: Click the Labs tab in the left side panel of the LMS. Copy or note the username and password that are
generated. Click the Launch Lab button. On the page that appears, enter the username and password in the
respective fields and click Login.
Thank You
© Simplilearn. All rights reserved.
