How to delete duplicates from a Pandas DataFrame?

This recipe helps you delete duplicates from a Pandas DataFrame

Recipe Objective

In many dataset we find many duplicate values so how to remove that.

So this is the recipe on how we can delete duplicates from a Pandas DataFrame.

Step 1 - Importing Library

import pandas as pd

We have only imported pandas which is needed.

Step 2 - Creating DataFrame

We have created a dataframe of which we will delete duplicate values. raw_data = {"first_name": ["Jason", "Jason", "Jason","Tina", "Jake", "Amy"], "last_name": ["Miller", "Miller", "Miller","Ali", "Milner", "Cooze"], "age": [42, 42, 1111111, 36, 24, 73], "preTestScore": [4, 4, 4, 31, 2, 3], "postTestScore": [25, 25, 25, 57, 62, 70]} df = pd.DataFrame(raw_data, columns = ["first_name", "last_name", "age", "preTestScore", "postTestScore"]) print(); print(df)

Step 3 - Removing Duplicate Values

We will drop duplicate values which will come after the first in the feature first name we will keep the last duplicate item. print(df.duplicated()) print(df.drop_duplicates(keep="first")) print(df.drop_duplicates(["first_name"], keep="last")) So the output comes as

  first_name last_name      age  preTestScore  postTestScore
0      Jason    Miller       42             4             25
1      Jason    Miller       42             4             25
2      Jason    Miller  1111111             4             25
3       Tina       Ali       36            31             57
4       Jake    Milner       24             2             62
5        Amy     Cooze       73             3             70

0    False
1     True
2    False
3    False
4    False
5    False
dtype: bool

  first_name last_name      age  preTestScore  postTestScore
0      Jason    Miller       42             4             25
2      Jason    Miller  1111111             4             25
3       Tina       Ali       36            31             57
4       Jake    Milner       24             2             62
5        Amy     Cooze       73             3             70

  first_name last_name      age  preTestScore  postTestScore
2      Jason    Miller  1111111             4             25
3       Tina       Ali       36            31             57
4       Jake    Milner       24             2             62
5        Amy     Cooze       73             3             70

Download Materials


What Users are saying..

profile image

Savvy Sahai

Data Science Intern, Capgemini
linkedin profile url

As a student looking to break into the field of data engineering and data science, one can get really confused as to which path to take. Very few ways to do it are Google, YouTube, etc. I was one of... Read More

Relevant Projects

Machine Learning Project to Forecast Rossmann Store Sales
In this machine learning project you will work on creating a robust prediction model of Rossmann's daily sales using store, promotion, and competitor data.

AI Video Summarization Project using Mixtral, Whisper, and AWS
In this AI Video Summarization Project, you will build a quiz generation tool by extracting key concepts from educational videos and generating concise summaries.

Build Regression Models in Python for House Price Prediction
In this Machine Learning Regression project, you will build and evaluate various regression models in Python for house price prediction.

Recommender System Machine Learning Project for Beginners-3
Content Based Recommender System Project - Building a Content-Based Product Recommender App with Streamlit

Deploy Transformer-BART Model on Paperspace Cloud
In this MLOps Project you will learn how to deploy a Tranaformer BART Model for Abstractive Text Summarization on Paperspace Private Cloud

Microsoft Fabric end-to-end Data Engineering project Overview
In this Microsoft Fabric project, you'll build a financial reporting agent that simplifies data management, automates analysis, and delivers real-time dashboards for wealth advisors and their clients.

Deep Learning Project for Text Detection in Images using Python
CV2 Text Detection Code for Images using Python -Build a CRNN deep learning model to predict the single-line text in a given image.

Build a Wealth Management Agentic AI Chatbot with MS Fabric
In this Agentic AI project , you will learn to build an intelligent financial assistant that autonomously analyzes your financial data, assesses risks, and designs personalized investment strategies, making wealth management more efficient and personalized to your needs

Build PowerBI Dashboard for Water Quality Sensor Data Analysis
In this PowerBI Project, you will learn to build a PowerBI Dashboard to analyze and visualize water quality sensor data from various European countries.

Medical Image Segmentation Deep Learning Project
In this deep learning project, you will learn to implement Unet++ models for medical image segmentation to detect and classify colorectal polyps.