Remove Duplicate Strings from a List in Python

How to Find Duplicates in a List - Python

Last Updated : 27 Nov, 2024

Finding duplicates in a list is a common task in programming. In Python, there are several ways to do this. Let’s explore the efficient methods to find duplicates.

Using a Set (Most Efficient for Large Lists)

Set() method is used to set a track seen elements and helps to identify duplicates.

Python

a = [1, 2, 3, 4, 5, 2, 6, 3]

# A set to keep track of elements that have been seen
seen = set()
# A list to store duplicates found in the input list
duplicates = []

# Iterate over each element in the list
for i in a:
    if i in seen:
        duplicates.append(i)
    else:
        seen.add(i)

print(duplicates)

Output

[2, 3]

There are more approaches to Find Duplicates in a List :

Table of Content

Using collection.counter
Using List Comprehension

Using collection.counter

Using collection.counter is efficient as in this Counter class from the collections module is greater for counting occurrences of each element .

Python

from collections import Counter

a = [1, 2, 3, 4, 5, 2, 6, 3]

# Use Counter to count 
#the occurrences of each element in the list
counts = Counter(a)
duplicates = [item for item, count in counts.items() if count > 1]
print(duplicates)

Output

[2, 3]

Using List Comprehension

List comprehension allows us to write more compact code. We then use a list comprehension to find items that appear more than once and store them in the duplicates list. Here’s how we can find duplicates using list comprehension.

Python

a = [1, 2, 3, 4, 5, 2, 6, 3]

# Create a list of duplicates using list comprehension
duplicates = [i for i in set(a) if a.count(i) > 1]
print(duplicates)

Output

[2, 3]

Remove Duplicate Strings from a List in Python

P

pragya22r4

Improve

Article Tags :

Practice Tags :

Similar Reads

Python - Remove Duplicates from a List

Removing duplicates from a list is a common operation in Python which is useful in scenarios where unique elements are required. Python provides multiple methods to achieve this. Using set() method is most efficient for unordered lists. Converting the list to a set removes all duplicates since sets

Remove Duplicate Strings from a List in Python

Removing duplicates helps in reducing redundancy and improving data consistency. In this article, we will explore various ways to do this. set() method converts the list into a set, which automatically removes duplicates because sets do not allow duplicate values.Pythona = ["Learn", "Python", "With"

Python | Get duplicate tuples from list

Sometimes, while working with records, we can have a problem of extracting those records which occur more than once. This kind of application can occur in web development domain. Let's discuss certain ways in which this task can be performed. Method #1 : Using list comprehension + set() + count() In

Python - Difference of Two Lists Including Duplicates

We are given two list we need find difference between two list. For example, a = [1, 2, 2, 3, 4] and b = [2, 3] we need to find difference of two list so that resultant output should be [1, 2, 4].Using a for loopA for loop can iterate through first list and remove matching elements from second list

Python Program to Find Duplicate sets in list of sets

Given a list of sets, the task is to write a Python program to find duplicate sets. Input : test_list = [{4, 5, 6, 1}, {6, 4, 1, 5}, {1, 3, 4, 3}, {1, 4, 3}, {7, 8, 9}]Output : [frozenset({1, 4, 5, 6}), frozenset({1, 3, 4})]Explanation : {1, 4, 5, 6} is similar to {6, 4, 1, 5} hence part of result.

Python | Altering duplicate values from given list

Many times we deal with the lists having identical numbers as a sequence and we wish to just keep the 1st occurrence of element and substituting all the occurrences with the different number. Let's discuss certain ways in which this can be done. Method #1 : Using list comprehension + enumerate() Thi

Python - Duplicate Element Indices in List

We are having a list we need to find the duplicate element indices. For example, we are given a list a = [10, 20, 30, 20, 40, 30, 50] we need to find indices of the duplicate items so that output should be {20: [1, 3], 30: [2, 5]}.Using a loop and a dictionaryWe iterate through the list using enumer

Python | Remove duplicates from nested list

The task of removing duplicates many times in the recent past, but sometimes when we deal with the complex data structure, in those cases we need different techniques to handle this type of problem. Let's discuss certain ways in which this task can be achieved. Method #1 : Using sorted() + set()Â Th

Python - Merging duplicates to list of list

We are given a list we need to merge the duplicate to lists of list. For example a list a=[1,2,2,3,4,4,4,5] we need to merge all the duplicates in lists so that the output should be [[2,2],[4,4,4]]. This can be done by using multiple functions like defaultdict from collections and Counter and variou

How to compare two lists in Python?

In Python, there might be a situation where you might need to compare two lists which means checking if the lists are of the same length and if the elements of the lists are equal or not. Let us explore this with a simple example of comparing two lists.Pythona = [1, 2, 3, 4, 5] b = [1, 2, 3, 4, 5] #