Data Science : Natural Language Processing ( SOC )

Experiment – 1

Demonstrate noise removal for any textual data and remove regular expression pattern such
as hash tag from textual data.

import re
def remove_noise(text):
# Remove hashtags
text = re.sub(r'#\w+', '', text)
# Remove other noise (e.g., special characters, URLs, numbers)
text = re.sub(r"[^a-zA-Z\s]", "", text) # Remove special characters
text = re.sub(r"\s+", " ", text) # Remove extra whitespaces
text = re.sub(r"http\S+|www\S+|https\S+", "", text) # Remove URLs
text = re.sub(r"\b\d+\b", "", text) # Remove numbers

return text.strip()

# Example usage
text = "Hello! This is a #sample text with #hashtags and some special characters!! 123 @acet.ac.in"
clean_text = remove_noise(text)
print(clean_text)

Output:

Hello This is a text with and some special characters acetacin

1
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 2

Perform lemmatization and stemming using python library nltk.

import nltk
from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')
nltk.download('omw-1.4')

lemmatizer=WordNetLemmatizer()
print(lemmatizer.lemmatize("running"))
print(lemmatizer.lemmatize("runs"))

def lemmatize(word):
lemmatizer=WordNetLemmatizer()
print("Verb Form: "+lemmatizer.lemmatize(word,pos="v"))
print("Noun Form: "+lemmatizer.lemmatize(word,pos="n"))
print("Adverb Form: "+lemmatizer.lemmatize(word,pos="r"))
print("Adjective Form: "+lemmatizer.lemmatize(word,pos="a"))
lemmatize('skewing')

Output:
running
run

Verb Form: skew

Noun Form: skewing
Adverb Form: skewing
Adjective Form: skewing

2
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

import nltk
from nltk.stem import PorterStemmer, LancasterStemmer
porter_stemmer=PorterStemmer()
print(porter_stemmer.stem('running'))
print(porter_stemmer.stem('runs'))
print(porter_stemmer.stem('ran'))

Output:
run
run
ran

3
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 3

Demonstrate object standardization such as replace social media slags from text.

slang_dict = {
"lol": "laughing out loud",
"omg": "oh my god",
"btw": "by the way",
"brb": "be right back",
"idk": "I don't know",
"tbh": "to be honest",
"imho": "in my humble opinion",
"afaik": "as far as I know",
"smh": "shaking my head",
"jk": "just kidding"
}

def standardize_text(text):
words = text.split()
standardized_words = []

for word in words:

if word.lower() in slang_dict:
standardized_words.append(slang_dict[word.lower()])
else:
standardized_words.append(word)

return ' '.join(standardized_words)

text = "lol that's so tbh idk why imho they would do that"
standardized_text = standardize_text(text)
print("Standardized text:", standardized_text)

Output:
Standardized text: laughing out loud that's so to be honest I don't know why in my humble opinion
they would do that

4
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 4

Perform part of speech tagging on any textual data.

import nltk
from nltk import word_tokenize, pos_tag
nltk.download('averaged_perceptron_tagger')
nltk.download('punkt')

def perform_pos_tagging(text):
tokens = nltk.word_tokenize(text)
tagged_tokens = nltk.pos_tag(tokens)
return tagged_tokens

# Example usage
text = "I love to explore new places and try different cuisines."
tagged_text = perform_pos_tagging(text)
print(tagged_text)

Output:

[('I', 'PRP'), ('love', 'VBP'), ('to', 'TO'), ('explore', 'VB'), ('new', 'JJ'), ('places', 'NNS'), ('and', 'CC'),
('try', 'VB'), ('different', 'JJ'), ('cuisines', 'NNS'), ('.', '.')]

5
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 5

Implement topic modeling using Latent Dirichlet Allocation (LDA) in python.

import gensim
from gensim import corpora

# Sample documents

documents = [
"Machine learning is a subset of artificial intelligence.",
"Python is a popular programming language for data science.",
"Natural language processing is used in many applications such as chatbots.",
"Topic modeling is a technique for extracting topics from text data."
]

# Tokenize and preprocess the documents

tokenized_docs = [doc.lower().split() for doc in documents]

# Create a dictionary from the tokenized documents

dictionary = corpora.Dictionary(tokenized_docs)

# Create a corpus (term-document frequency)

corpus = [dictionary.doc2bow(doc) for doc in tokenized_docs]

# Build the LDA model

num_topics = 2 # Number of topics to extract
lda_model = gensim.models.LdaModel(corpus, num_topics=num_topics, id2word=dictionary,
passes=10)

# Print the extracted topics and their top words

for idx, topic in lda_model.print_topics(num_words=5):
print(f"Topic {idx + 1}: {topic}")

# Get the topic distribution for a sample document

sample_doc = "Machine learning and data science go hand in hand."
sample_doc_bow = dictionary.doc2bow(sample_doc.lower().split())
sample_doc_topics = lda_model.get_document_topics(sample_doc_bow)
print(f"\nSample Document Topics: {sample_doc_topics}")

Output:

Topic 1: 0.056*"language" + 0.056*"is" + 0.055*"such" + 0.055*"applications" + 0.055*"many"

6
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Topic 2: 0.080*"a" + 0.080*"is" + 0.057*"for" + 0.034*"data." + 0.034*"topics"

Sample Document Topics: [(0, 0.30881903), (1, 0.691181)]

7
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 6

Demonstrate Term Frequency – Inverse Document Frequency ( TF – IDF ) using python.

!pip install scikit-learn

import nltk
nltk.download('punkt')

from sklearn.feature_extraction.text import TfidfVectorizer

# Sample documents
documents = [
"Machine learning is a subset of artificial intelligence.",
"Python is a popular programming language for data science.",
"Natural language processing is used in many applications such as chatbots.",
"Topic modeling is a technique for extracting topics from text data."
]

# Create the TF-IDF vectorizer

vectorizer = TfidfVectorizer()

# Fit the vectorizer to the documents and transform the documents into TF-IDF features
tfidf_matrix = vectorizer.fit_transform(documents)

# Get the feature names (terms)

feature_name = vectorizer.get_feature_names_out()

# Print the TF-IDF matrix

print("TF-IDF Matrix:")
print(tfidf_matrix.toarray())

# Print the TF-IDF values for each term in each document

print("\nTF-IDF Values:")
for doc_index, doc in enumerate(documents):
print(f"Document {doc_index + 1}:")
for term_index, term in enumerate(feature_name):
tfidf_value = tfidf_matrix[doc_index, term_index]
if tfidf_value > 0:
print(f"{term}: {tfidf_value:.4f}")

Output:

8
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

TF-IDF Values:
Document 1:
artificial: 0.3993
intelligence: 0.3993
is: 0.2084
learning: 0.3993
machine: 0.3993
of: 0.3993
subset: 0.3993
Document 2:
data: 0.3183
for: 0.3183
is: 0.2106
language: 0.3183
popular: 0.4037
programming: 0.4037
python: 0.4037
science: 0.4037
Document 3:
applications: 0.3179
as: 0.3179
chatbots: 0.3179
in: 0.3179
is: 0.1659
language: 0.2507
many: 0.3179

9
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

natural: 0.3179
processing: 0.3179
such: 0.3179
used: 0.3179
Document 4:
data: 0.2702
extracting: 0.3427
for: 0.2702
from: 0.3427
is: 0.1788
modeling: 0.3427
technique: 0.3427
text: 0.3427
topic: 0.3427
topics: 0.3427

10
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 7

Demonstrate Word Embeddings using word2vec.

!pip install gensim

from gensim.models import Word2Vecec

# Step 1: Prepare training data (list of tokenized sentences)

sentences = [
["machine", "learning", "is", "fun"],
["deep", "learning", "uses", "neural", "networks"],
["natural", "language", "processing", "is", "a", "part", "of", "AI"],
["word2vec", "creates", "word", "embeddings"],
["AI", "is", "the", "future"],
]

# Step 2: Train Word2Vec model

model = Word2Vec(sentences, vector_size=50, window=3, min_count=1, workers=2, sg=1)

# Step 3: Use the model

# Get embedding vector for a word

word_vector = model.wv["learning"]
print("Vector for 'learning':\n", word_vector)

# Find similar words

print("\nMost similar words to 'AI':")
similar = model.wv.most_similar("AI", topn=3)
for word, score in similar:
print(f"{word}: {score:.4f}")

11
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 8

Implement text classification using naïve bayes classifier and text blob library.

!pip install -U scikit-learn

!pip install -U textblob

import nltk
nltk.download('punkt')

from textblob import TextBlob

from textblob.classifiers import NaiveBayesClassifier

# Sample training data

train_data = [
('I love this car.', 'positive'),
('This view is amazing.', 'positive'),
('I feel great!', 'positive'),
('I dislike this product.', 'negative'),
('This place is horrible.', 'negative'),
('I feel sad.', 'negative')
]

# Create the Naive Bayes classifier

classifier = NaiveBayesClassifier(train_data)

# Sample test data

test_data = [
'I like this movie.',
'This food is terrible.',
'I am happy.'
]

# Classify the test data

for text in test_data:
sentiment = classifier.classify(text)
print(f'Text: {text}')
print(f'Sentiment: {sentiment}\n')

Output:

Text: I like this movie.

Sentiment: positive

12
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Text: This food is terrible.

Sentiment: positive

Text: I am happy.
Sentiment: positive

13
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 9

Apply support vector machine for text classification.

!pip install -U scikit-learn

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, classification_report

# Sample data
documents = [
("I love natural language processing.", "positive"),
("Machine learning is fascinating.", "positive"),
("Python is widely used in data science.", "positive"),
("I dislike noisy environments.", "negative"),
("This movie is terrible.", "negative"),
("I feel sad today.", "negative")
]

# Split the data into features and labels

texts, labels = zip(*documents)

# Create the TF-IDF vectorizer

vectorizer = TfidfVectorizer()

# Transform the text data into TF-IDF features

features = vectorizer.fit_transform(texts)

# Split the data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2, random_state=42)

# Initialize the SVM classifier

svm_classifier = SVC()

# Train the classifier

svm_classifier.fit(X_train, y_train)

# Make predictions on the test set

y_pred = svm_classifier.predict(X_test)

# Evaluate the performance of the classifier

accuracy = accuracy_score(y_test, y_pred)
report = classification_report(y_test, y_pred)

print("Accuracy:", accuracy)

14
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

print("\nClassification Report:")
print(report)

Output:
Accuracy: 0.0

Classification Report:
precision recall f1-score support

negative 0.00 0.00 0.00 0.0

positive 0.00 0.00 0.00 2.0
accuracy 0.00 2.0
macro avg 0.00 0.00 0.00 2.0
weighted avg 0.00 0.00 0.00 2.0

15
Aditya College of Engineering & Technology
 Data Science : Natural Language Processing ( SOC )

Experiment – 10

Convert text to vectors ( using term frequency ) and apply cosine similarity to provide
closeness among two text.

from sklearn.feature_extraction.text import CountVectorizer

from sklearn.metrics.pairwise import cosine_similarity

# Sample documents
documents = [
"I love natural language processing.",
"Machine learning is fascinating.",
"Python is widely used in data science."
]

# Initialize the CountVectorizer

vectorizer = CountVectorizer()

# Fit and transform the documents to obtain the term frequency (TF) vectors
tf_vectors = vectorizer.fit_transform(documents).toarray()

# Calculate the cosine similarity between two documents

doc1 = tf_vectors[0]
doc2 = tf_vectors[1]
similarity = cosine_similarity([doc1], [doc2])[0][0]

print(f"Text 1: {documents[0]}")
print(f"Text 2: {documents[1]}")
print(f"Cosine Similarity: {similarity:.4f}")

Output:

Text 1: I love natural language processing.

Text 2: Machine learning is fascinating.
Cosine Similarity: 0.0000

16
Aditya College of Engineering & Technology