APEX INSTITUTE OF TECHNOLOGY

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

MACHINE LEARNING (21CSH-286)

Faculty: Prof. (Dr.) Vineet Mehan (E13038)

Lecture – 4 DISCOVER . LEARN . EMPOWER

1
Data Pre-Processing
Machine Learning: Course Objectives
COURSE OBJECTIVES
The Course aims to:
1. Understand and apply various data handling and visualization techniques.
2. Understand about some basic learning algorithms and techniques and their applications, as well as
general questions related to analysing and handling large data sets.
3. To develop skills of supervised and unsupervised learning techniques and implementation of these to
solve real life problems.
4. To develop basic knowledge on the machine techniques to build an intellectual machine for making
decisions behalf of humans.
5. To develop skills for selecting suitable model parameters and apply them for designing optimized
machine learning applications.

2
COURSE OUTCOMES

On completion of this course, the students shall be able to:-

CO2 Understand data pre-processing techniques and apply these for data cleaning.

3
 Unit-1 Syllabus
Unit-1 Introduction to Machine Learning
Introduction to Definition of Machine Learning, Working principles of Machine
Machine Learning Learning; Classification of Machine Learning algorithms: Supervised
Learning, Unsupervised Learning, Reinforcement Learning, Semi-
Supervised Learning; Applications of Machine Learning.
Data Pre- Data Sourcing and Cleaning, Handling Missing data, Encoding
Processing and Categorical data, Feature Scaling, Handling Time Series data; Feature
Feature Selection techniques, Data Transformation, Normalization,
Extraction Dimensionality reduction
Data Visualization Data Frame Basics, Different types of analysis, Different types of
plots, Plotting fundamentals using Matplotlib, Plotting Data
Distributions using Seaborn.

4
SUGGESTIVE READINGS
• TEXT BOOKS:
• There is no single textbook covering the material presented in this course. Here is a list of books
recommended for further reading in connection with the material presented:
• T1: Tom.M.Mitchell, “Machine Learning, McGraw Hill International Edition”.
• T2: Ethern Alpaydin,” Introduction to Machine Learning. Eastern Economy Edition, Prentice Hall of
India, 2005”.
• T3: Andreas C. Miller, Sarah Guido, Introduction to Machine Learning with Python, O’REILLY (2001).

• REFERENCE BOOKS:
• R1 Sebastian Raschka, Vahid Mirjalili, Python Machine Learning, (2014)
• R2 Richard O. Duda, Peter E. Hart, David G. Stork, “Pattern Classification, Wiley, 2nd Edition”.
• R3 Christopher Bishop, “Pattern Recognition and Machine Learning, illustrated Edition, Springer, 2006”.

5
Data Sourcing
• For data sourcing Panda is used.

• Panda is a python Library for analyzing data.

• Name?
• Panda = Panel Data + Python Data Analysis (Combination) gave the
name.
• Panel data is a subset of longitudinal data where observations are for
the same subjects each time.
By: Prof. (Dr.) Vineet Mehan 6
Data Sourcing
• Use of Panda ?

• Pandas allow us to analyze big data and make conclusions based on

statistical theories.

• Pandas can clean messy data sets, and make them readable and
relevant.

• Pandas are used in Data Science.

By: Prof. (Dr.) Vineet Mehan 7
Data Sourcing
• Data Science: is a branch of computer science where we study how to
store, use and analyze data for deriving information from it.

• How to install Pandas?

• 1. Open cmd prompt
• 2. Type
• >>> python –m pip install pandas

By: Prof. (Dr.) Vineet Mehan 8

Make a data Frame that tells the type of
vehicles that passed a toll plaza.
• import pandas
• mydataset = { 'cars': ["Maruti", "Hundai", "Tata"], 'passings': [20, 12,
15]}
• myvar = pandas.DataFrame(mydataset)
• print(myvar)

By: Prof. (Dr.) Vineet Mehan 9

Import pandas as pd and use pd

By: Prof. (Dr.) Vineet Mehan 10

Read data from a CSV File

By: Prof. (Dr.) Vineet Mehan 11

Reading CSV but print without converting to
string

By: Prof. (Dr.) Vineet Mehan 12

Checking the pandas version

By: Prof. (Dr.) Vineet Mehan 13

Pandas Data Frames
• A Pandas DataFrame is a 2 dimensional data structure, like a 2
dimensional array, or a table with rows and columns.

• Create a simple Panda Data Frame

By: Prof. (Dr.) Vineet Mehan 14

Load the CSV file into data Frame

By: Prof. (Dr.) Vineet Mehan 15

Data Cleaning
• Data cleaning means fixing bad data in your data set.

• Bad data could be:

• Empty cells

• Data in wrong format

• Wrong data

• Duplicates

By: Prof. (Dr.) Vineet Mehan 16

 The data set contains some empty cells ("Date" in row
22, and "Calories" in row 18 and 28).

By: Prof. (Dr.) Vineet Mehan 17

 The data set contains wrong format ("Date" in row 26).

By: Prof. (Dr.) Vineet Mehan 18

 The data set contains wrong data ("Duration" in row 7).

By: Prof. (Dr.) Vineet Mehan 19

 The data set contains duplicates (row 11 and 12).

By: Prof. (Dr.) Vineet Mehan 20

1. Remove Rows
• One way to deal with empty cells is to remove rows that contain
empty cells.

• This is usually OK, since data sets can be very big, and removing a few
rows will not have a big impact on the result.

• See Row 17 and 27 (removed)

By: Prof. (Dr.) Vineet Mehan 21

Pandas dropna() method allows the user to analyze
and drop Rows/Columns with Null values

By default, the dropna() method returns a new

DataFrame, and will not change the original.

By: Prof. (Dr.) Vineet Mehan 22

By default, the dropna() method returns a new
DataFrame, and will not change the original.

If you want to change the original DataFrame, use

the inplace = True argument.

By: Prof. (Dr.) Vineet Mehan 23

3. Replace Empty Values

See Row 17 replaced with 130

The fillna() method allows us to replace

empty cells with a value.

It will Replace NULL values with the number 130.

By: Prof. (Dr.) Vineet Mehan 24

4. Replace value in a particular column

Values are replaced at position 17, 27, 91,

118, and 141 in the Calories column only.

By: Prof. (Dr.) Vineet Mehan 25

5. Replace Using Mean, Median, or Mode
• A common way to replace empty cells, is to calculate the mean,
median or mode value of the column.

• Mean  Average

• Median  Center value

• Mode  Most common occurring value

By: Prof. (Dr.) Vineet Mehan 26

Empty Values are replaced with mean
at position 17, 27, 91, 118, and 141 in
the Calories column only.

Mean here is 375.790244

By: Prof. (Dr.) Vineet Mehan 27

Empty Values are replaced with median
at position 17, 27, 91, 118, and 141 in
the Calories column only.

Median here is 318.6

By: Prof. (Dr.) Vineet Mehan 28

Empty Values are replaced with mode
at position 17, 27, 91, 118, and 141 in
the Calories column only.

Mode here is 300.0

By: Prof. (Dr.) Vineet Mehan 29

Wrong Data
• "Wrong data" does not have to be "empty cells" or "wrong format", it
can just be wrong, like if someone registered "199" instead of "1.99".

• Sometimes you can spot wrong data by looking at the data set,
because you have an expectation of what it should be.

• If you take a look at our data set, you can see that in row 7, the
duration is 450, but for all the other rows the duration is between 30
and 60.

By: Prof. (Dr.) Vineet Mehan 30

By: Prof. (Dr.) Vineet Mehan 31
 One way to fix wrong values is to
replace them with something else.

In our example, it is most likely a typo,

and the value should be "45" instead of
"450", and we could just insert "45" in
row 7:

By: Prof. (Dr.) Vineet Mehan 32

For Larger Data
• For small data sets you might be able to replace the wrong data one
by one, but not for big data sets.

• To replace wrong data for larger data sets you can create some rules,
e.g. set some boundaries for legal values, and replace any values that
are outside of the boundaries.

By: Prof. (Dr.) Vineet Mehan 33

By: Prof. (Dr.) Vineet Mehan 34
Removing Rows
• Another way of handling wrong data is to remove the rows that
contains wrong data.

• This way you do not have to find out what to replace them with, and
there is a good chance you do not need them to do your analyses.

• Value at position no 7 is removed

By: Prof. (Dr.) Vineet Mehan 35

By: Prof. (Dr.) Vineet Mehan 36
Duplicate Data
• Duplicate rows are rows that have been registered more than one
time.

• By taking a look at our test data set, we can assume that row 11 and
12 are duplicates.

• To discover duplicates, we can use the duplicated() method.

• The duplicated() method returns a Boolean values for each row.

By: Prof. (Dr.) Vineet Mehan 37
Above program Returns True for every
row that is a duplicate, otherwise False

By: Prof. (Dr.) Vineet Mehan 38

Removing Duplicates
• To remove duplicates, use the drop_duplicates() method.

The duplicate row (row no 12) is now removed

By: Prof. (Dr.) Vineet Mehan 39

Summary
• Methods of Sourcing Data

• Methods of Cleaning Data

40
Task
• Applying various methods that are used for sourcing the data by
taking a suitable arrays\datasets etc. (BT-Level3)

• Design a model that is used to clean Empty cells, Data in wrong

format, Wrong data, and Duplicates. (BT-Level6)

By: Prof. (Dr.) Vineet Mehan 41

REFERENCES
• https://www.javatpoint.com/machine-learning

• https://www.tutorialspoint.com/machine_learning/index.htm

• https://www.w3schools.com/python/

42
THANK YOU

For queries
Email: [email protected]
43