2/24/2020 3.
2_filtering - Jupyter Notebook
Filtering
In [1]:
import numpy as np
import pandas as pd
In [2]:
# Create a Dictionary
d = {
'Name':['Amarend','Ajay','Preety','Rakesh','Raju','Shyam',
'Kiran','Rishi','Prem','Raj','Ravina','Premjit'],
'Exam':['Semester 1','Semester 1','Semester 1','Semester 1','Semester 1','Semester 1',
'Semester 2','Semester 2','Semester 2','Semester 2','Semester 2','Semester 2'],
'Subject':['Mathematics','Mathematics','Mathematics','Science','Science','Science',
'Mathematics','Mathematics','Mathematics','Science','Science','Science'],
'Score':[62,47,55,74,31,77,85,63,42,67,89,81]}
# Create a dataframe
df = pd.DataFrame(d,columns=['Name','Exam','Subject','Score'])
df
Out[2]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
1 Ajay Semester 1 Mathematics 47
2 Preety Semester 1 Mathematics 55
3 Rakesh Semester 1 Science 74
4 Raju Semester 1 Science 31
5 Shyam Semester 1 Science 77
6 Kiran Semester 2 Mathematics 85
7 Rishi Semester 2 Mathematics 63
8 Prem Semester 2 Mathematics 42
9 Raj Semester 2 Science 67
10 Ravina Semester 2 Science 89
11 Premjit Semester 2 Science 81
View a column of the dataframe in pandas python:
localhost:8888/notebooks/Machine Learning/Python/3.2_filtering.ipynb 1/7
2/24/2020 3.2_filtering - Jupyter Notebook
In [5]:
df['Name']
Out[5]:
0 Amarend
1 Ajay
2 Preety
3 Rakesh
4 Raju
5 Shyam
6 Kiran
7 Rishi
8 Prem
9 Raj
10 Ravina
11 Premjit
Name: Name, dtype: object
View two or more columns of the dataframe in pandas:
In [18]:
df[['Name', 'Score']]
Out[18]:
Name Score
0 Amarend 62
1 Ajay 47
2 Preety 55
3 Rakesh 74
4 Raju 31
5 Shyam 77
6 Kiran 85
7 Rishi 63
8 Prem 42
9 Raj 67
10 Ravina 89
11 Premjit 81
View first two rows of the dataframe in pandas:
localhost:8888/notebooks/Machine Learning/Python/3.2_filtering.ipynb 2/7
2/24/2020 3.2_filtering - Jupyter Notebook
In [6]:
df[:2]
Out[6]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
1 Ajay Semester 1 Mathematics 47
In [7]:
df.head(2)
Out[7]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
1 Ajay Semester 1 Mathematics 47
View last two rows of the dataframe in pandas:
In [20]:
df[-2:]
Out[20]:
Name Exam Subject Score
10 Ravina Semester 2 Science 89
11 Premjit Semester 2 Science 81
Filter pandas dataframe by column value
Method 1 : DataFrame Way
localhost:8888/notebooks/Machine Learning/Python/3.2_filtering.ipynb 3/7
2/24/2020 3.2_filtering - Jupyter Notebook
In [21]:
# based on one condition
df1 = df[df['Score']>60]
df1
Out[21]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
3 Rakesh Semester 1 Science 74
5 Shyam Semester 1 Science 77
6 Kiran Semester 2 Mathematics 85
7 Rishi Semester 2 Mathematics 63
9 Raj Semester 2 Science 67
10 Ravina Semester 2 Science 89
11 Premjit Semester 2 Science 81
In [22]:
# based on multiple conditions
df1A = df[(df['Score']>60) & (df['Subject']=='Mathematics')]
df1B = df[(df.Score>60) & (df.Subject=='Mathematics')]
#df1A
df1B
Out[22]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
6 Kiran Semester 2 Mathematics 85
7 Rishi Semester 2 Mathematics 63
In [31]:
# Select only a few columns under some conditions
df1C = df[(df.Score>60) & (df.Subject=='Mathematics')][['Name','Score']]
df1C
Out[31]:
Name Score
0 Amarend 62
6 Kiran 85
7 Rishi 63
Method 2 : Query Function
In pandas package, there are multiple ways to perform filtering. The above code can also be written like the
code shown below. This method is elegant and more readable and you don't need to mention dataframe name
localhost:8888/notebooks/Machine Learning/Python/3.2_filtering.ipynb 4/7
2/24/2020 3.2_filtering - Jupyter Notebook
everytime when you specify columns (variables).
In [33]:
df2 = df.query('Score > 60 & Subject == "Mathematics"')
df2
Out[33]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
6 Kiran Semester 2 Mathematics 85
7 Rishi Semester 2 Mathematics 63
Method 3 : loc function
loc is an abbreviation of location term. All these 3 methods return same output. It's just a different ways of doing
filtering rows.
In [36]:
df3 = df.loc[(df.Score>60) & (df.Subject=='Mathematics')]
df3
Out[36]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
6 Kiran Semester 2 Mathematics 85
7 Rishi Semester 2 Mathematics 63
Difference between loc and iloc function
loc considers rows based on index labels. Whereas iloc considers rows based on position in the index so it only
takes integers. Let's create a sample data for illustration
localhost:8888/notebooks/Machine Learning/Python/3.2_filtering.ipynb 5/7
2/24/2020 3.2_filtering - Jupyter Notebook
In [38]:
x = pd.DataFrame({"col1" : np.arange(1,20,2)}, index=[9,8,7,6,0, 1, 2, 3, 4, 5])
x
Out[38]:
col1
9 1
8 3
7 5
6 7
0 9
1 11
2 13
3 15
4 17
5 19
iloc - Index Position
In [39]:
x.iloc[0:5]
Out[39]:
col1
9 1
8 3
7 5
6 7
0 9
loc - Index Label
localhost:8888/notebooks/Machine Learning/Python/3.2_filtering.ipynb 6/7
2/24/2020 3.2_filtering - Jupyter Notebook
In [40]:
x.loc[0:5]
Out[40]:
col1
0 9
1 11
2 13
3 15
4 17
5 19
Note : x.loc[0:5] returns 6 rows (inclusive of 5 which is 6th element)
It is because loc does not produce output based on index position. It considers labels of index only which can
be alphabet as well and includes both starting and end point. Refer the example below.
In [41]:
# more examples - (offline) Data Analytics - Preprocessing 4
In [3]:
df.head()
Out[3]:
Name Exam Subject Score
0 Amarend Semester 1 Mathematics 62
1 Ajay Semester 1 Mathematics 47
2 Preety Semester 1 Mathematics 55
3 Rakesh Semester 1 Science 74
4 Raju Semester 1 Science 31
In [9]:
#df.sortby('Name')
In [ ]:
localhost:8888/notebooks/Machine Learning/Python/3.2_filtering.ipynb 7/7
