Natural Language Processing

Q1. A company has a dataset containing raw customer reviews. Design an NLP pipeline to
preprocess this data for sentiment analysis. Discuss each step in detail.
Example Data
"The product was fantastic! But delivery was delayed."
"Horrible customer service. Would not recommend!"
"Great quality for the price."

Solution:
The NLP pipeline can help in pre-processing the raw customer reviews for sentiment analysis so
that the data will be structured and ready to be modeled correctly. A further explanation of the
following steps is given below:
1. Text Cleaning:
o The text is made standard by removing special characters, punctuation, numbers,
and extra whitespace. This step of standardization reduces the noise that could
mislead the sentiment model.
o Example: "The product was fantastic! But delivery was delayed." → "The product
was fantastic But delivery was delayed"
2. Tokenization:
o Split the text into smaller units (tokens), typically words or sentences, for easier
processing. Tokenizing sentences can aid in capturing contextual sentiment for
compound sentences.
o Example: "The product was fantastic But delivery was delayed" → ["The", "product",
"was", "fantastic", "But", "delivery", "was", "delayed"]
3. Lowercasing:
o Convert all text to lowercase to ensure that identical words in different cases are
treated uniformly.
o Example: ["The", "product", "was", "fantastic"] → ["the", "product", "was",
"fantastic"]
4. Stopword Removal:
o Remove frequent words (like "the," "is," "and") that do not contribute to sentiment.
This reduces noise and enhances focus on sentiment-carrying words.
o Example: ["the", "product", "was", "fantastic"] → ["product", "fantastic"]
5. Stemming or Lemmatization:
o Convert words to their base or root forms to reduce dimensionality. Lemmatization
is preferred for sentiment analysis since it retains meaning.
o Example: ["delayed", "fantastic"] → ["delay", "fantastic"]
6. Feature Extraction:
 o Transform textual data into numerical format. Use techniques like TF-IDF for
traditional approaches or embeddings like Word2Vec/BERT for context-sensitive
representations. Embeddings are particularly effective for capturing sentiment
nuances in phrases like "not bad."
7. Sentiment Labeling:
o Assign a sentiment score or label (positive, negative, or neutral) based on patterns in
the preprocessed data, typically using a supervised model trained on annotated
examples.
This pipeline will prepare data such as "The product was fantastic! But delivery was delayed."
into a structured format like ["product", "fantastic", "delivery", "delay"] and numerical
embeddings for downstream sentiment analysis.

Q2. Write a Python function to tokenize and remove stop words from the “The quick brown
fox jumps over the lazy dog.”. Explain how this step affects the quality of an NLP model.
Solution:
Python Function for Tokenization and Stopword Removal
Below is a Python function that tokenizes the input sentence and removes stopwords using the
NLTK library:

Q3.Compare and contrast traditional feature extraction techniques like TF-IDF with modern
embeddings like Word2Vec and BERT. Discuss the impact of these advancements.
You are tasked with classifying emails as spam or non-spam. Justify the choice of feature
extraction technique you would use and why.
Solution:
Comparing TF-IDF, Word2Vec, and BERT
Traditional technique, TF-IDF, generates sparse vectors by checking the word importance based
on frequency. Though computationally quite simple, there are some big disadvantages because
the semantic and contextual aspects of the meaning are beyond its consideration potential.
Hence, it would not fit more complex NLP tasks. On the other side, Word2Vec—modern
embedding—captures the semantic relationship of words, such as "King" and "Queen." It fails
for out-of-vocabulary words, and its embeddings do not consider context. BERT is the state-of-
the-art approach that builds contextual embeddings considering sentence-level meaning, thus it
can handle polysemy and subtle shades of meaning; however, it is resource-intensive in terms of
computation.
Justification for Spam Classification
For spam classification, the choice of feature extraction depends on the size, complexity, and
computational resources of a dataset:
 TF-IDF works well for resource-constrained or smaller datasets. This is interpretable, fast,
and integrates well with traditional models like SVM or logistic regression.
 Word2Vec probably does better when the size of the dataset is a moderately large and
semantic understanding of words e.g., "free" and "offer" are important.
 BERT may be suited to large datasets or cases where subtlety like the contextual meaning of
the word "urgent" across different phrases needs to be captured. These embeddings, fed
into deep learning classifiers, achieve state-of-the-art results, but may be computationally
expensive.

Q4. Write a Python script to train a Recurrent Neural Network (RNN) on the Shakespeare text
dataset available at this link. Follow the NLP pipeline to train the model and generate
Shakespearean-style text.
Solution:
Recurrent Neural Networks (RNNs) are designed for sequential data, making them ideal for text
generation tasks. When applied to the Shakespeare text dataset, an RNN learns patterns and
structures in the text by processing it in sequences and predicting the next character or word.
Following the NLP pipeline, the dataset is preprocessed, tokenized, and used to train the model.
Once trained, the RNN can generate Shakespearean-style text by predicting and appending
tokens sequentially, capturing the unique style and flow of the original works. Below is the link of
model: RNN MODEL LINK