How to delete duplicates from a Pandas DataFrame?
This recipe helps you delete duplicates from a Pandas DataFrame
Recipe Objective
In many dataset we find many duplicate values so how to remove that.
So this is the recipe on how we can delete duplicates from a Pandas DataFrame.
Table of Contents
Step 1 - Importing Library
import pandas as pd
We have only imported pandas which is needed.
Step 2 - Creating DataFrame
We have created a dataframe of which we will delete duplicate values.
raw_data = {"first_name": ["Jason", "Jason", "Jason","Tina", "Jake", "Amy"],
"last_name": ["Miller", "Miller", "Miller","Ali", "Milner", "Cooze"],
"age": [42, 42, 1111111, 36, 24, 73],
"preTestScore": [4, 4, 4, 31, 2, 3],
"postTestScore": [25, 25, 25, 57, 62, 70]}
df = pd.DataFrame(raw_data, columns = ["first_name", "last_name", "age",
"preTestScore", "postTestScore"])
print(); print(df)
Step 3 - Removing Duplicate Values
We will drop duplicate values which will come after the first in the feature first name we will keep the last duplicate item.
print(df.duplicated())
print(df.drop_duplicates(keep="first"))
print(df.drop_duplicates(["first_name"], keep="last"))
So the output comes as
first_name last_name age preTestScore postTestScore
0 Jason Miller 42 4 25
1 Jason Miller 42 4 25
2 Jason Miller 1111111 4 25
3 Tina Ali 36 31 57
4 Jake Milner 24 2 62
5 Amy Cooze 73 3 70
0 False
1 True
2 False
3 False
4 False
5 False
dtype: bool
first_name last_name age preTestScore postTestScore
0 Jason Miller 42 4 25
2 Jason Miller 1111111 4 25
3 Tina Ali 36 31 57
4 Jake Milner 24 2 62
5 Amy Cooze 73 3 70
first_name last_name age preTestScore postTestScore
2 Jason Miller 1111111 4 25
3 Tina Ali 36 31 57
4 Jake Milner 24 2 62
5 Amy Cooze 73 3 70
What Users are saying..
Ray han
I think that they are fantastic. I attended Yale and Stanford and have worked at Honeywell,Oracle, and Arthur Andersen(Accenture) in the US. I have taken Big Data and Hadoop,NoSQL, Spark, Hadoop... Read More