Tokenizing Strings in List of Strings - Python

The task of tokenizing strings in a list of strings in Python involves splitting each string into smaller units, known as tokens, based on specific delimiters. For example, given the list a = ['Geeks for Geeks', 'is', 'best computer science portal'], the goal is to break each string into individual words or tokens, resulting in a list of lists: [['Geeks', 'for', 'Geeks'], ['is'], ['best', 'computer', 'science', 'portal']].

Using list comprehension

List comprehension is a concise way of creating lists. It allows for looping over an iterable and applying operations or expressions to generate new lists. When combined with split() this provides a very efficient way to tokenize strings.

a = ['Geeks for Geeks', 'is', 'best computer science portal']

res = [sub.split() for sub in a]
print(res)

[['Geeks', 'for', 'Geeks'], ['is'], ['best', 'computer', 'science', 'portal']]

Explanation: list comprehension iterates over each string in a, applying the split() method to split each string into a list of words.

Using resplit()

For more complex tokenization, where delimiters are not just spaces, re.split() from the re module can be used. It allows us to split strings based on regular expressions, making it suitable for handling multiple delimiters, such as spaces, punctuation, and other special characters.

import re
a = ['Geeks for Geeks', 'is', 'best computer science portal']

res = [re.split(r'\s+', sub) for sub in a]
print(res)

[['Geeks', 'for', 'Geeks'], ['is'], ['best', 'computer', 'science', 'portal']]

Explanation: list comprehension splits each string in a into words using the re.split() with the pattern r'\s+', which matches one or more whitespace characters.

Using map()

map() applies a given function to all items in an iterable . It’s a functional programming tool that is often used in Python to avoid explicit loops. By combining map() with split() , we can tokenize strings efficiently in a functional programming style.

a = ['Geeks for Geeks', 'is', 'best computer science portal']

res = list(map(str.split, a))
print(res)

[['Geeks', 'for', 'Geeks'], ['is'], ['best', 'computer', 'science', 'portal']]

Explanation: map() applies split() to each string in a, splitting them into words and then the result is converted back to a list.

manjeet_04

Follow

Improve

Article Tags :

• Python
• Python Programs
• Python list-programs

Practice Tags :

• python