Data Science with Python

Lesson 7— Data Manipulation with Pandas

 What You Will Learn

Pandas and its features

Different data structures of Pandas

Creating Series and DataFrame with data inputs

Viewing, selecting, and accessing elements in a data structure

Handling vectorized operations

Learning how to handle missing values

Analyzing data with different data operation methods

 Why Pandas

NumPy is great for mathematical computing. Then why do we need Pandas?

Pandas with several

functionalities

NumPy
 Why Pandas

NumPy is great for mathematical computing. Then why do we need Pandas?

Intrinsic data
alignment

Data Structures
Data operation
handling major
functions
use cases
Pandas

Data standardization
functions ? Functions for handling
missing data
.
 Pandas Features

The various features of Pandas makes it an efficient library for Data Scientists.

Powerful data
structure

Fast and efficient

High performance
data wrangling
merging and joining
of data sets

Pandas
Intelligent and Easy data aggregation
automated data and transformation
alignment

Tools for reading/

writing data
 Data Structures

The four main libraries of Pandas data structure are:

• One-dimensional labeled array

Series
• Supports multiple data types
• Two-dimensional labeled array
• Supports multiple data types
Data Frame
• Input can be a Series
• Three-dimensional labeled array • Input can be another DataFrame
• Supports multiple data types
• Items  axis 0 Panel
• Major axis  rows • Four-dimensional labeled array
• Minor axis columns • Supports multiple data types
Panel 4D • Labels  axis 0
(Experimental) • Items  axis 1
• Major axis  rows
• Minor axis columns
 Understanding Series

Series is a one-dimensional array-like object containing data and labels (or index).

Data 4 11 21 36
0 1 2 3

Label(index)

Data alignment is intrinsic and will not be broken until changed explicitly by program.
 Series

Series can be created with different data inputs:

Data Input

• Integer
• ndarray 2 3 8 4
• String
• dict
• Python Object 0 1 2 3
• scalar
• Floating Point
• list Label(index)

Data Types
Series
 How to Create Series

Key points to note while creating a series are as follows:

• Import Pandas as it is the main library
• Apply the syntax and pass the data elements as arguments
• Import NumPy while working with ndarrays

Basic Method

4 11 21 36
S = pd.Series(data, index = [index])
Series
 Create Series from List
This example shows you how to create a series from a list:

Import libraries

Pass list as an argument

Data value

Index

Data type

We have not created index for data but notice that data alignment is done automatically
 Create Series from ndarray

This example shows you how to create a series from an ndarray:

ndarray for countries

Pass ndarray as an argument

countries
Index

Data type
 Create Series from dict

A series can also be created with dict data input for faster operations.
dict for countries and their gdp

Countries have been passed as an index

and GDP as the actual data value

GDP

Country

Data type
 Create Series from Scalar

Scalar input

Index

Data

index

Data type
 Accessing Elements in Series
Data can be accessed through different functions like loc, iloc by passing data element position or
index range.
 Vectorized Operations in Series

Vectorized operations are performed by the data element’s position.

Add the series

Vectorized Operations in Series
Knowledge Check
KNOWLEDGE How is an index for data elements assigned while creating a Pandas series ? Select all
CHECK that apply.

a. Created automatically

b. Needs to be assigned

c. Once created can not be changed or altered

d. Index is not applicable as series is one-dimensional

 How is an index for data elements assigned while creating a Pandas series ? Select all
KNOWLEDGE that apply.
CHECK

a.
Created automatically

b. Needs to be assigned

c. Once created can not be changed or altered

d.
Index is not applicable as series is one-dimensional

The correct answer is a, b .

Explanation: Data alignment is intrinsic in Pandas data structure and happens automatically. One can also assign index to data
elements.
KNOWLEDGE
What will the result be in vector addition if label is not found in a series?
CHECK

a. Marked as Zeros for missing labels

b. Labels will be skipped

c. Marked as NaN for missing labels

d. Will throw an exception, index not found

KNOWLEDGE
CHECK
What will the result be in vector addition if label is not found in a series?

a.
Marked as Zeros for missing labels

b. Labels will be skipped

c. Marked as NaN for missing labels

d.
Will throw an exception, index not found

The correct answer is c .

Explanation: The result will be marked as NaN (Not a Number) for missing labels.
 DataFrame

DataFrame is a two-dimensional labeled data structure with columns of potentially different types.

Data Input

• Integer
• ndarray 2 3 8 4
• String
• dict 5 8 10 1
• Python Object
• scalar
• Floating Point 0 1 2 3
• list
Label(index)

Data Types
DataFrame
 Create DataFrame from Lists
Let’s see how you can create a DataFrame from lists:

Pass the list to the DataFrame

 Create DataFrame from dict
This example shows you how to create a DataFrame from a series of dicts:

dict one dict two

Entire dict
 View DataFrame

You can view a DataFrame by referring the column name or with the describe function.
Create DataFrame from dict of Series
Create DataFrame from ndarray

Create an ndarrays with years

Create a dict with the ndarray

Pass this dict to a new DataFrame

Create DataFrame from DataFrame

Create a DataFrame from a

DataFrame
Demo 01—View and Select Data
Demonstrate how to view and select data in a DataFrame.
 Missing Values

Various factors may lead to missing data values:

Data not provided by the

source Software issue Data integration issue Network issue
 Handle Missing Values
It’s difficult to operate on a dataset when it has missing values or uncommon indices.
 Handle Missing Values with Functions
The dropna function drops all the values with uncommon indices.
 Handle Missing Values with Functions

The fillna function fills all the uncommon indices with a number instead of dropping them.

Fill the missing values with zero

Handle Missing Values with Functions- Example
 Data Operation
Data operation can be performed through various built-in methods for faster data processing.
 Data Operation with Functions
While performing data operation, custom functions can be applied with the applymap method.

Declare a custom function

Test the function

Apply the function to the DataFrame

 Data Operation with Statistical Functions
This example shows data operations with different statistical functions.

Create a DataFrame with two test

Apply the max function to find the

maximum score

Apply the mean function to find

the average score

Apply the std function to find the standard

deviation for both the tests
 Data Operation Using Groupby
This example shows how to operate data using the groupby function.

Create a DataFrame with first and

last name as former presidents

Group the DataFrame with the first name

Group the DataFrame with the first name

 Data Operation – Sorting
This example shows how to sort data

Sort values by first name

Demo 02—Data Operations
Demonstrate how to perform data operations.
 Data Standardization
This example shows how to standardize a dataset.
Create a function to return the standardize value

Apply the function to the entire dataset

Standardized test data is applied for the entire

DataFrame
Knowledge Check
KNOWLEDGE
CHECK
What is the result of DataFrame[3:9]?

a. Series with sliced index from 3 to 9

b. dict of index position 3 and index position 9

c. DataFrame of sliced rows index from 3 to 9

d. DataFrame with data elements at index 3 to index9

KNOWLEDGE
CHECK
What is the result of DataFrame[3:9]?

a. Series with sliced index from 3 to 9

b. dict of index position 3 and index position 9

c. DataFrame of sliced rows index from 3 to 9

d. DataFrame with data elements at index 3 to index9

The correct answer is . c

Explanation: This is DataFrame slicing technique with indexing or selection on data elements. When a user
passes the range 3:9, the entire range from 3 to 9 gets sliced and displayed as output.
KNOWLEDGE
CHECK
What does the fillna() method do?

a. Fills all NaN values with zeros

b. Fills all NaN values with one

c. Fills all NaN values with values mentioned in the parenthesis

d. Drops NaN values from the dataset

KNOWLEDGE
CHECK
What does the fillna() method do?

a. Fills all NaN values with zeros

b. Fills all NaN values with One

c. Fills all NaN values with values mentioned in the parenthesis

d. Drops NaN values from the dataset

The correct answer is . c

Explanation: fillna is one of the basic methods to fill NaN values in a dataset with a desired value by passing
that in parenthesis.
 File Read and Write Support

read_hdf
read_excel to_hdf read_clipboard
to_excel to_clipboard

read_csv read_html
to csv to_html

read_json read_pickle
to_json to_pickle

read_sql read_stata
read_sas
to_sql to_stata
to sas
Pandas SQL operation
Pandas SQL operation
Pandas SQL operation
 Activity—Sequence it Right!
The code here is buggy. You have to correct its sequence to debug it. To do that, click any two code
snippets, which you feel are out of place, to swap their places.

Click any two code snippets to swap them.

Assignment
Assignment
Quiz
QUIZ
Which of the following data structures is used to store three-dimensional data?
1

a. Series

b. DataFrame

c. Panel

d. PanelND
 QUIZ
Which of the following data structures is used to store three-dimensional data?
1

a. Series

b. DataFrame

c. Panel

d. PanelND

The correct answer is c.

Explanation: Panel is a data structure used to store three-dimensional data.

QUIZ
Which method is used for label-location indexing by label?
2

a. iat

b. iloc

c. loc

d. std
 QUIZ
Which method is used for label-location indexing by label?
2

a. iat

b. iloc

c. loc

d. std

The correct answer is c.

Explanation: The loc method is used to for label-location indexing by label; iat is strictly integer location and
iloc is integer-location-based indexing by position.
QUIZ
While viewing a dataframe, head() method will .
3

a. return only the first row

b. return only headers or column names of the DataFrame

c. return the first five rows of the DataFrame

d. throw an exception as it expects parameter(number) in parenthesis

 QUIZ
While viewing a dataframe, head() method will .
3

a. return only the first row

b. return only headers or column name of the DataFrame

c. return the first five rows of the DataFrame

d. throw an exception as it expects parameter(number) in parenthesis

The correct answer is c.

Explanation: The default value is 5 if nothing is passed in head method. So it will return the first five rows
of the DataFrame.
 Key Takeaways

Let us take a quick recap of what we have learned in the lesson:

Pandas is an open source library for data analysis and is an efficient

data wrangling tool in Python.
The four main libraries of Pandas are Series, DataFrame, Panel, and
Panel 4D.
DataFrame is a two-dimensional labeled data structure with columns
of potentially different data types.
To access data elements in a series, 'loc' and 'iloc' methods can be
used.
 Key Takeaways

The 'iat' method enables selection of elements in a DataFrame by

index position and returns the corresponding data element.
Missing data values in Pandas can be resolved through
two built-in methods such as dropna and fillna.

Pandas supports multiple files for data analysis such as

Excel, PyTables, Clipboard, HTML, pickle, dta, SAS, SQL,
JSON, and CSV.
 This concludes “Data Manipulation with Pandas.”
The next lesson is “Machine Learning with SciKit Learn.”