How to Find & Drop duplicate columns in a Pandas DataFrame?

Last Updated : 22 Dec, 2023

Let’s discuss How to Find and drop duplicate columns in a Pandas DataFrame. First, Let’s create a simple Dataframe with column names 'Name', 'Age', 'Domicile', and 'Age'/'Marks'.

Find Duplicate Columns from a DataFrame

To find duplicate columns we need to iterate through all columns of a DataFrame and for each and every column it will search if any other column exists in DataFrame with the same contents already. If yes then that column name will be stored in the duplicate column set. In the end, the function will return the list of column names of the duplicate column. In this way, we can find duplicate labels in Pandas.

Python3

import pandas as pd

def getDuplicateColumns(df):

    # Create an empty set
    duplicateColumnNames = set()

    # Iterate through all the columns of dataframe
    for x in range(df.shape[1]):

        # Take column at xth index.
        col = df.iloc[:, x]

        # Iterate through all the columns
        for y in range(x + 1, df.shape[1]):

            # Take column at yth index.
            otherCol = df.iloc[:, y]

            # Check if two columns at x & y
            if col.equals(otherCol):
                duplicateColumnNames.add(df.columns.values[y])

    return list(duplicateColumnNames)


# Driver code
if __name__ == "__main__":

    # List of Tuples
    students = [
        ('Ankit', 34, 'Uttar pradesh', 34),
        ('Riti', 30, 'Delhi', 30),
        ('Aadi', 16, 'Delhi', 16),
        ('Riti', 30, 'Delhi', 30),
        ('Riti', 30, 'Delhi', 30),
        ('Riti', 30, 'Mumbai', 30),
        ('Ankita', 40, 'Bihar', 40),
        ('Sachin', 30, 'Delhi', 30)
    ]

    # Create a DataFrame object
    df = pd.DataFrame(students, columns=['Name', 'Age', 
                                     'Domicile', 'Marks'])

    # Get list of duplicate columns
    duplicateColNames = getDuplicateColumns(df)

    for column in duplicateColNames:
        print('Column Name : ', column)

Output:

Column Name:  Marks

Remove Duplicate Columns from a DataFrame

Below are the ways by which we can remove duplicate labels in Pandas in Python:

Using drop_duplicates()
Using df.loc[]
Using df.columns.duplicated()
Using df.drop

Drop duplicate columns from a DataFrame using drop_duplicates()

Pandas drop_duplicates() method helps in removing duplicates from the Pandas Dataframe In Python.

Python3

# Drop duplicate columns
df2 = df.T.drop_duplicates().T
print(df2)

Output:

     Name Age       Domicile
0   Ankit  34  Uttar pradesh
1    Riti  30          Delhi
2    Aadi  16          Delhi
3    Riti  30         Mumbai
4  Ankita  40          Bihar
5  Sachin  30          Delhi

Remove duplicate columns from a DataFrame using df.loc[]

Pandas df.loc[] attribute access a group of rows and columns by label(s) or a boolean array in the given DataFrame.

Python3

# Remove duplicate columns pandas DataFrame
df2 = df.loc[:,~df.columns.duplicated()]
print(df2)

Output:

     Name Age       Domicile
0   Ankit  34  Uttar pradesh
1    Riti  30          Delhi
2    Aadi  16          Delhi
3    Riti  30         Mumbai
4  Ankita  40          Bihar
5  Sachin  30          Delhi

Remove duplicate columns from a DataFrame using df.columns.duplicated()

Pandas df.duplicated() method helps in analyzing duplicate values only. It returns a boolean series which is True only for Unique elements.

Python3

# Use DataFrame.columns.duplicated() to drop duplicate columns
duplicate_cols = df.columns[df.columns.duplicated()]
df.drop(columns=duplicate_cols, inplace=True)
print(df)

Output:

     Name       Domicile
0   Ankit  Uttar pradesh
1    Riti          Delhi
2    Aadi          Delhi
3    Riti         Mumbai
4  Ankita          Bihar
5  Sachin          Delhi

Drop duplicate columns in a DataFrame using df.drop

To remove the duplicate columns we can pass the list of duplicate column names returned by our user defines function getDuplicateColumns() to the Dataframe.drop() method.

Python3

# import pandas library
import pandas as pd

def getDuplicateColumns(df):

    # Create an empty set
    duplicateColumnNames = set()
    
    # Iterate through all the columns
    # of dataframe
    for x in range(df.shape[1]):
        
        # Take column at xth index.
        col = df.iloc[:, x]
        
        for y in range(x + 1, df.shape[1]):
            
            # Take column at yth index.
            otherCol = df.iloc[:, y]
            
            if col.equals(otherCol):
                duplicateColumnNames.add(df.columns.values[y])
                
    # Return list of unique column names
    # whose contents are duplicates.
    return list(duplicateColumnNames)

# Driver code
if __name__ == "__main__" :

    # List of Tuples
    students = [
            ('Ankit', 34, 'Uttar pradesh', 34),
            ('Riti', 30, 'Delhi', 30),
            ('Aadi', 16, 'Delhi', 16),
            ('Riti', 30, 'Delhi', 30),
            ('Riti', 30, 'Delhi', 30),
            ('Riti', 30, 'Mumbai', 30),
            ('Ankita', 40, 'Bihar', 40),
            ('Sachin', 30, 'Delhi', 30)
        ]

    # Create a DataFrame object
    df = pd.DataFrame(students,
                        columns =['Name', 'Age', 'Domicile', 'Marks'])

    # Dropping duplicate columns
    rslt_df = df.drop(columns = getDuplicateColumns(df))

    print("Resultant Dataframe :")

    # Show the dataframe
rslt_df