Remove Duplicate Strings from a List in Python

Removing duplicates helps in reducing redundancy and improving data consistency. In this article, we will explore various ways to do this. set() method converts the list into a set, which automatically removes duplicates because sets do not allow duplicate values.

a = ["Learn", "Python", "With", "GFG", "GFG"]  

# Remove duplicates using set  
s = list(set(a))  

print(s)  

['with', 'GFG', 'Python', 'Learn']

Explanation:

• The list is converted into a set to eliminate duplicates.
• The set is converted back into a list.

Note: The order of elements changes when using set() because a set in Python is an unordered collection of unique elements. Sets do not preserve the order of items as they appear in the original list. Instead, the elements are stored in a way that ensures fast lookups, which can rearrange the order.

Let's explore some more methods and see how we can duplicate strings from a list in Python.

Using dict.fromkeys()

dict.fromkeys() eliminates duplicates and preserves the order of elements in the list. The keys retain their order as dictionaries in Python maintain insertion order.

# List of strings with duplicates  
a = ["Learn", "Python", "with", "GFG", "GFG"]  

# Remove duplicates while preserving order  
s = list(dict.fromkeys(a))  

print(s)  

['Learn', 'Python', 'with', 'GFG']

Explanation:

• dict.fromkeys() creates a dictionary where keys are unique elements from the list.

Using a for Loop with a Set

This method iterates through the list while using a set to track already seen elements. By manually checking for duplicates, it ensures that each element is added to the result only once. Since the for loop processes elements sequentially, the original order is preserved.

a = ["Learn", "Python", "with", "GFG", "GFG"]  

# Remove duplicates manually  
b = []  
seen = set()  

for word in a:  
    if word not in seen:  
        b.append(word)  
        seen.add(word)  

print(b)

['Learn', 'Python', 'with', 'GFG']

Explanation:

• A set seen is used to track already encountered elements.
• Only elements not present in seen are added to the 'b' list.

Using List Comprehension

List comprehension can be combined with a set or a temporary list to remove duplicates. During iteration, it checks if the current element has already been added to a new list, ensuring each element is only added once.

a = ["Learn", "Python", "with", "GFG", "GFG"]  

# Remove duplicates using list comprehension  
b = []  
[b.append(word) for word in a if word not in b]  

print(b)   

['Learn', 'Python', 'with', 'GFG']

Explanation:

• List comprehension iterates through the list and appends elements to a new list only if they are not already present.