Prof. V.V.

Subrahmanyam
School of Computer and Information Sciences
Indira Gandhi National Open University (IGNOU)
New Delhi
Date: 22nd Aug, 2024 Time : 4-00PM to 4-30PM
 Text Mining
 Text mining, also known
as text data mining, is the
process of transforming
unstructured text into
a structured format to
identify meaningful
patterns and new insights.
 Text Preprocessing - Introduction
 Text data derived from natural language is
unstructured and noisy.
 So text preprocessing is a critical step to
transform messy, unstructured text data
into a form that can be effectively used to
train machine learning models, leading to
better results and insights.
 Text Preprocessing
 Text preprocessing refers to a series of
techniques used to clean, transform and
prepare raw textual data into a format that
is suitable for natural language processing
(NLP) or Text Mining or Machine Learning
(ML) tasks.
 Goal of Text Preprocessing
 The goal of text preprocessing is to
enhance the quality and usability of
the text data for subsequent analysis
or modeling.
Common Text Preprocessing / Cleaning Steps
 Lower Casing  Conversion of
 Removal of Punctuations emoticons to words
 Removal of Stopwords  Conversion of emojis to
 Removal of Frequent words words
 Removal of Rare words  Removal of URLs
 Stemming
 Removal of HTML tags
 Lemmatization
 Removal of emojis  Chat words conversion
 Removal of emoticons  Spelling correction
 Lower Casing
 Lower casing is a common text preprocessing
technique. The idea is to convert the input text
into same casing format so that, for example 'text',
'Text' and 'TEXT' are treated the same way.
 This is more helpful for text featurization
techniques like frequency, tfidf as it helps to
combine the same words together thereby
reducing the duplication and get correct counts /
tfidf values.
 Removal of Punctuations
 This is again a text standardization process
that will help to treat 'hurray' and 'hurray!'
in the same way.
 We also need to carefully choose the list of
punctuations to exclude depending on the
use case.
 Removal of Stopwords
 Stopwords are commonly occuring words in a language
like 'the', 'a' and so on.
 They can be removed from the text most of the times,
as they don't provide valuable information for
downstream analysis.
 In cases like Part of Speech(POS) tagging, we should
not remove them as provide very valuable information
about the POS.
 Removal of Stop Frequent Words
 In the previos preprocessing step, we observed the
stopwords based on language information. But
say, if we have a domain specific corpus, we
might also have some frequent words which are of
not so much importance to us.
 So this step is to remove the frequent words in the
given corpus. If we use something like tfidf, this is
automatically taken care of.
Some of the Domain Specific Corpus
Frequent Words….

 I, us, DM, Help, We, Hi,

Please, Get, Thanks etc..
 Removal of Rare Words
 This is very similar to previous
preprocessing step but we will remove the
rare words from the corpus.
 We can combine all the list of words
(stopwords, frequent words and rare
words) and create a single list to remove
them at once.
 Stemming
 Stemming is the process of reducing inflected or derived
words to their word stem, base or root form.
 For example, if there are two words in the
corpus walks and walking, then stemming will stem the
suffix to make them walk.
 But say in another example, we have two
words console and consoling, the stemmer will remove
the suffix and make them consol which isn’t a proper
English word.
Contd…
 There are several type of stemming algorithms
available and one of the famous one is porter stemmer
which is widely used.
 Porter stemmer is for English language. If we are
working with other languages, we can use snowball
stemmer.
Stemming
 Example
 We can see that words like private and propose have
their e at the end chopped off due to stemming. This is
not intented.
 What can we do for that? We can use Lemmatization
in such cases.
 Lemmatization

 Lemmatization is similar to stemming in reducing

inflected words to their word stem but differs in the
way that it makes sure the root word (also called as
lemma) belongs to the language.
Examples: Propose, Private
Illustration of Lemmatization and Stemming
 Removal of Emojis
 With more and more usage of social media
platforms, there is an explosion in the usage
of emojis in our day to day life as well.
Probably we might need to remove these
emojis for some of our textual analysis.
 Removal of Emoticons

 There is a minor difference between emojis and

emoticons.
 Emoticon is built from keyboard characters that when
put together in a certain way represent a facial
expression, an emoji is an actual image.
 :-) is an emoticon
 😀 is an emoji
 Conversion of Emoticon to Words
 In the previous step, we have removed the emoticons.
In case of use cases like sentiment analysis, the
emoticons give some valuable information and so
removing them might not be a good solution. What
can we do in such cases?
 One way is to convert the emoticons to word format so
that they can be used in downstream modeling
processes.
 Conversion of Emoji to Words

 Now let us do the same for Emojis as well.

 We may make use of a dictionary to convert the emojis
to corresponding words.
 Again this conversion might be better than emoji
removal for certain use cases. Please use the one that is
suitable for the use case.
 Removal of URLs
 Next preprocessing step is to remove any URLs present
in the data.
 For example, if we are doing a X (Twitter) analysis,
then there is a good chance that the tweet will have
some URL in it. Probably we might need to remove
them for our further analysis.
 Removal of HTML Tags
 One another common preprocessing technique that
will come handy in multiple places is removal of
HRML tags.
 This is especially useful, if we scrap the data from
different websites. We might end up having html
strings as part of our text.
 Chat Words Conversion

 This is an important text preprocessing step if we are

dealing with chat data.
 People do use a lot of abbreviated words in chat and
so it might be helpful to expand those words for our
analysis purposes.
 Examples
 AFAIK=As Far As I Know
 AFK=Away From Keyboard
 ASAP=As Soon As Possible
 ATK=At The Keyboard
 ATM=At The Moment
 A3=Anytime, Anywhere, Anyplace
 BAK=Back At Keyboard
 BBL=Be Back Later
 BBS=Be Back Soon
 BFN=Bye For Now
 B4N=Bye For Now
 Spelling Correction

 One another important text preprocessing step is

spelling correction.
 Typos are common in text data and we might want to
correct those spelling mistakes before we do our
analysis.
 Tokenization
 Tokenization is the process of breaking up
text into separate tokens, which can be
individual words, phrases, or whole
sentences.
 In some cases, punctuation and special
characters (symbols like %, &, $) are
discarded in the process.
Tokenization
Contd…
A few common operations that require tokenization
include:
 Finding how many words or sentences appear in text
 Determining how many times a specific word or
phrase exists
 Accounting for which terms are likely to co-occur
 Parts of Speech (POS) Tagging
 This is one of the more advanced text preprocessing
technique.
 This step augments the input text with additional
information about the sentence’s grammatical structure.
 Each word is, therefore, inserted into one of the predefined
categories such as a noun, verb, adjective, etc.
 This step is also sometimes referred to as grammatical
tagging.
 Term Frequency
 Term frequency tells you how much a term occurs in
a document.
 Terms can be either individual words or phrases
containing multiple words.
 Since documents differ in length, it’s possible that a
term would appear more times in longer documents
than shorter ones.
Contd…
 Thus, you can calculate term frequency by dividing the
number of times the term appears, by the total
number of terms in the document, as a way of
normalization.
 Term Frequency = [Number of times the term appears
in the document] / [Total number of terms in the
document]
While Working with Python Language….
 We will be using the NLTK (Natural Language Toolkit)

# import the necessary libraries

import nltk
import string
import re
To Remove Punctuation
To remove white space
 THANK YOU
Email: [email protected]