Introduction to Pandas Library in Python
Introduction to Pandas
1. Objective
• Getting started with pandas
• Introduction to panda’s data structure
• NumPy vs pandas
• Series vs Data frames
• Creating, Reading and Writing
• Indexing, Selecting and assigning data to data frames
• Grouping and sorting
• Handling missing values
• Use fundamental Python data frames for data science
2. Getting Started with pandas:
In the previous lab, we dove into detail on NumPy and its functionalities, which provides
efficient storage and manipulation of dense typed arrays in Python. Here we’ll build on this
knowledge by looking in detail at the data structures provided by the Pandas library. Pandas is a
package built on top of NumPy, and provides an efficient implementation of a Series and
DataFrame. In this lab, we will focus on the mechanics of using Series, DataFrame, and related
structures effectively.
Just as we generally import NumPy under the alias np, we will import Pandas under the alias pd:
import pandas as pd
This important convention will be used through the lab.
2.1 Introduction to pandas Data Structures
At the very basic level, Pandas objects can be thought of as enhanced versions of NumPy
structured arrays in which the rows and columns are identified with labels rather than simple
integer indices. To get started with pandas, you will need to get comfortable with its two workhorse
data structures: Series and DataFrame. While they are not a universal solution for every problem,
they provide a solid, easy-to-use basis for most applications.
Installing Pandas library
pip install pandas # for regular python environments
conda install pandas # for anaconda environment
!pip install pandas # for jupytor notebook (applicable in our case)
By: Faizan Irshad Page 2
Introduction to Pandas
2.2 Series
❖ A Pandas Series is a one-dimensional array of indexed data. It can be created from a
list or array as follows:
Example:
data = pd.Series([0.25, 0.5, 0.75, 1.0])
print(data)
Series wraps both a sequence of values and a sequence of indices, which we can access with the
values and index attributes.
print(data.values) //output: [0.25 0.5 0.75 1. ]
Like with a NumPy array, data can be accessed by the associated index via the familiar
Python square-bracket notation:
print(data[1]) //output: 0.5
For example, if we wish, we can use strings as an index:
data = pd.Series([0.25, 0.5, 0.75, 1.0], index=['a', 'b', 'c', 'd'])
print(data)
output:
and the item access works as expected: data[‘b’] = ?
Important: We can even use noncontiguous or nonsequential indices:
data = pd.Series([0.25, 0.5, 0.75, 1.0], index=[2, 5, 3, 7])
By: Faizan Irshad Page 3
Introduction to Pandas
2.3 DataFrame
A DataFrame represents a tabular, spreadsheet-like data structure containing an ordered
collection of columns, each of which can be a different value type (numeric, string,
Boolean, etc.). The DataFrame has both a row and column index; it can be thought of as a
dict of Series.
2.4 Reading Data files
Being able to create a DataFrame or Series by hand is handy. But, most of the time, we
won't actually be creating our own data by hand. Instead, we'll be working with data that
already exists.
Data can be stored in any of a number of different forms and formats. By far the most
basic of these is the humble CSV file. CSV file is a table of values separated by commas.
Hence the name: "Comma-Separated Values", or CSV.
Loading file from C/users folder
data = pd.read_csv("covid19.csv")
load file using google drive if using colab
from google.colab import drive
drive.mount('/content/drive')
import pandas as pd
dataset = pd.read_csv('/content/drive/MyDrive/covid19.csv')
Loading from any other specific location on computer, use
backslashes for address
data = pd.read_csv("C:\\Users\\ABPKSUP\\Desktop\\AI Enabled Data
Analytics -LUMS\\Class 4, Pandas Library\\covid19.csv")
We can use the shape attribute to check how large the resulting DataFrame is:
print(data.shape)
We can examine the contents of the resultant DataFrame using the head() command,
which grabs the first five rows:
print(data.head())
By: Faizan Irshad Page 4
Introduction to Pandas
3. Access in Data Frame
In Python, we can access the property of an object by accessing it as an attribute. A book object, for
example, might have a title property, which we can access by calling book Title. Columns in a pandas
DataFrame work in much the same way.
print(data.Country)
Moreover, we can also access data frame specific value as:
print(data['Country’][2])
4. Indexing in pandas
The indexing operator and attribute selection are nice because they work just like they do in the rest
of the Python ecosystem. As a novice, this makes them easy to pick up and use. However, pandas
has its own accessor operators, loc and iloc. For more advanced operations, these are the ones
you're supposed to be using:
4.1 Index-based selection
Pandas indexing works in one of two paradigms. The first is index-based selection:
selecting data based on its numerical position in the data. iloc follows this paradigm.
To select the first row of data in a DataFrame, we may use the following:
print(data.iloc[0])
To get a column with iloc, we can do the following:
print(data.iloc[:,0])
By: Faizan Irshad Page 5
Introduction to Pandas
To select the column from just the first, second, and third row, we would do:
print(data.iloc[:3,0])
It's also possible to pass a list: print(data.iloc[[0,2,5],0])
Finally, it's worth knowing that negative numbers can be used in selection. This will start counting
forwards from the end of the values. So, for example here are the last five elements of the dataset.
print(data.iloc[-5:])
By: Faizan Irshad Page 6
Introduction to Pandas
4.2 Label-based selection
The second paradigm for attribute selection is the one followed by the loc operator: label-
based selection. In this paradigm, it's the data index value, not its position, which matters.
loc is label-based, means that you have to specify rows and columns based on their row
and column labels. iloc is integer position-based, so you have to specify rows and columns by
their integer position values (0-based integer position). Since your dataset usually has
meaningful indices, it's usually easier to do things using loc instead. For example, here's one
operation that's much easier using loc.
print(data.loc[:, ['ObservationDate', 'Province/State', 'Country']])
5. Conditional Selection
So far we've been indexing various strides of data, using structural properties of the DataFrame
itself. To do interesting things with the data, however, we often need to ask questions based on
conditions.
For example, suppose that we're interested specifically in Anhui province cases
print(data.loc[data['Province/State'] == 'Anhui'])
/ / total 494 records
We can use the OR (|) to bring the two questions together( for AND you can use &):
print(data.loc[(data['Province/State'] == 'Anhui') | (data['Deaths'] ==
0)])
total 32,893 records found.
Pandas comes with a few built-in conditional selectors, two of which we will highlight here.
The first is isin which lets you select data whose value "is in" a list of values. For example case in
in Peru or Spain.
print(data.loc[(data.Country.isin(['Peru','Spain']))])
The second is isnull or isna (and its companion notnull). These methods let you highlight
values which are (or are not) empty (NaN).
print(data.loc[(data.Country.notnull())])
print(data.loc[(data['Province/State'].notnull())])
By: Faizan Irshad Page 7
Introduction to Pandas
To find records with null values
print(data.loc[(data['Province/State'].isnull())])
print(data.loc[(data['Province/State'].isna())])
By: Faizan Irshad Page 8
Introduction to Pandas
Assigning data:
Going the other way, assigning data to a DataFrame is easy.
Adding a New Column to the data frame with selected values..
data['New Column'] = 'dummy_values'
print(data['New Column'])
6. Functions and maps
To see a list of unique values we can use the unique() function:
print(data.Country.unique())
More functions which are available with pandas (here pd means pandas
library while df is the name of our dataset like we used name ‘data’ above.
It can be any name that we assign to our data)
By: Faizan Irshad Page 9
Introduction to Pandas
By: Faizan Irshad Page 10
Introduction to Pandas
Important:
Writing to files
df.describe() | Summary statistics for numerical columns
df.mean() | Returns the mean of all columns
df.corr() | Returns the correlation between columns in a DataFrame
df.count() | Returns the number of non-null values in each DataFrame column
df.max() | Returns the highest value in each column
df.min() | Returns the lowest value in each column
df.median() | Returns the median of each column
df.std() | Returns the standard deviation of each column
pd.to_datetime | To convert column format to date time
Combining:
Concatenate:
The concat function is used to concatenate two or more DataFrames along a particular axis (either rows or
columns).
df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]})
# Concatenate along rows (axis=0)
result = pd.concat([df1, df2], axis=0)
print(result)
Output
A B
0 1 3
1 2 4
0 5 7
1 6 8
By: Faizan Irshad Page 11
Introduction to Pandas
# Concatenate along columns
result2 = pd.concat([df1, df2], axis=1)
print(result2)
OUTPUT
A B A B
0 1 3 5 7
1 2 4 6 8
Merge:
The merge function is used to merge two DataFrames based on a common column or index.
df1 = pd.DataFrame({'key': ['A', 'B'], 'value': [1, 2]})
df2 = pd.DataFrame({'key': ['A', 'B'], 'value': [3, 4]})
# Merge based on a common column 'key'
result = pd.merge(df1, df2, on='key')
Output
key value_x value_y
0 A 1 3
1 B 2 4
By: Faizan Irshad Page 12
Introduction to Pandas
Practice Task 1
Create a DataFrame fruits that looks like this:
Bananas Apples
0 20 17
Practice Task 2
Create a DataFrame sales that looks like below:
Bananas Apples
September 200 120
October 165 90
Practice Task 3
Create a series marks that looks like:
7.1 Practice Task 4
Read the following csv dataset of Groceries given to you into a DataFrame called grocer
7.2 Practice Task 5
Select the item description column from grocer and assign the result to the variable item desc.
7.3 Practice Task 6
Select the first value from the member_number column of grocer and assign it to variable
first_member.
7.4 Practice Task 7
Select the first 20 rows of data and save it to variable first_rows.
By: Faizan Irshad Page 13
Introduction to Pandas
7.5 Practice Task 8
Create a variable selected_grocer containing the values of all columns only for purchase made in
2015.
7.6 Practice Task 9
Create a variable containing the member_number and date columns of the first 500 records.
7.7 Practice Task 10
Create a DataFrame having purchases both in 2014 and 2015 year.
By: Faizan Irshad Page 14
