Text Classification

Assoc. Prof. Dr. Nguyen Ngoc Vu

14/01/2024 1
Decomposing Texts with Bag
of Words Model

2
Basic Concepts
Topic Overview: Introduction to the Bag of Words (BoW)
model, a foundational method in textual data
representation and language processing.
Functionality of BoW:
Unordered Collection: Treats each document as a collection of
words without considering their order.
Focus on Frequency: Concentrates on the frequency or occurrence
of each word, ignoring grammar and word order.
Feature Extraction: BoW extracts features based only on
word multiplicity, disregarding positional data and
grammatical rules.
Example Illustration: Sentences like “The cat sat on the
mat” and “The mat sat on the cat” have identical
representations due to ignoring word order.
Importance in NLP: Serves as a foundational basis for
various text processing and language-related tasks.
Limitations: Lack of sensitivity to syntax and semantics,
leading to potential loss of meaning and context.

14/01/2024 3
 Vector Representation
Core Concept: BoW model represents documents as frequency vectors.
Vector Definition in Mathematics: Traditionally, an object with magnitude and direction,
used to represent physical entities.
Vector Transformation in NLP: In the BoW model, vectors represent the frequency of
words in a document, not physical entities.
Frequency Vector Explained:
Structure: An organized list where each slot corresponds to a word from a predetermined
vocabulary.
Content: The value indicates the frequency of the corresponding word in a specific document.
Dictionary Construction:
Purpose: Consists of unique words gleaned from a corpus, each linked to a unique index.
Impact: The size of the dictionary determines the length and detail of the frequency vectors.
BoW Process Visualization:
Example: Converting the sentence “The cat sat on the mat” into a frequency vector [2, 1, 1, 1,
1].
Advantages of BoW:
Simplicity: Transforms variable-length textual data into fixed-length vectors, compatible with
machine learning algorithms.
Limitations of BoW:
Ignores Word Sequence: Loses out on semantics and contextual relationships between words.
High-Dimensional Data: An extensive dictionary can lead to very high-dimensional vectors.

14/01/2024 4
 Advanced Concepts and Enhancements to the
Bag of Words Model
Concept Overview: The BoW model emphasizes word frequency over grammar and
word order, providing an overview of dominant text themes.
Frequency-Based Approach: Focuses on identifying prevalent words to infer
primary subjects or themes within the document.
Hypothetical Scenario: Distinguishing between documents by observing dominant
words like “cat” and “play” versus “dog” and “run.”
Strengths:
Simplicity: Transforms text into concise lists of words and frequencies, facilitating easy
computational processing.
Efficiency: Enables rapid text analysis, ideal for processing large volumes of text quickly.
Limitations:
Bypasses Nuances: Overlooks the inherent meanings, contextual relevance, and
grammatical constructs of language.
Lacks Depth: Does not capture specific emotions or emphases conveyed by unique word
sequencing.
Impact: Despite limitations, the BoW model's ability to summarize primary themes
makes it a valuable analytical tool.

14/01/2024 5
 Implementation in Text Classification
Text Classification Using BoW: Overview of the 3-step process to transform text
into a computationally understandable format.
Step 1: Tokenization:
Description: Breaking a sentence into individual words.
Example: “The cat sat on the mat” becomes “The”, “cat”, “sat”, “on”, “the”, “mat”.
Step 2: Creating a Vocabulary:
Purpose: Compile a list of all unique words from the sentences.
Outcome: A vocabulary like “The”, “cat”, “sat”, “on”, “the”, “mat” from the given
sentence.
Step 3: Transforming Documents into Feature Vectors:
Process: Use the vocabulary to convert each sentence into a list of numbers
representing word occurrences.
Illustration: “The cat sat on the mat” translates to the feature vector [1, 1, 1, 1, 1, 1].
Complex Example: “The cat sat on the cat” becomes [1, 2, 1, 1, 1, 0], indicating word
frequencies.
Application: Feature vectors enable computers to classify and understand
sentence content.
Python Libraries: Utilizing NLTK for tokenization and scikit-learn for creating
BoW models enhances ease and efficiency.

14/01/2024 6
Bayesian Text Classification:
The Naive Approach

7
 Basic Concepts
Thomas Bayes: An English statistician, philosopher, and Presbyterian minister
known for Bayes’ theorem.
Bayes' Theorem: Describes the probability of an event based on prior
knowledge of related conditions, allowing for updated probabilities with new
evidence.
Naive Bayes Classifier:
Assumption: Independence among predictors, treating each feature as unrelated to
any other.
Applications: Suited for high-dimensional datasets and used in text classification,
spam filtering, and sentiment analysis.
Effectiveness: Despite the simplistic assumption, it is effective and a popular baseline
method.
Understanding Naive Bayes:
Scenario: Detective tool analogy - identifying types of candy based on color, shape,
and size.
Process: Calculates chances of a type based on individual clues, considering each
separately for simplicity and speed.
Strengths: Works well with a variety of clues and can classify different types of text.
Math Trick: Utilizes Bayes' theorem to calculate the most likely type and solve the
classification problem.

14/01/2024 8
 Understanding the Mathematics behind Naive Bayes
Bayes' Theorem: A mathematical formula for making updated guesses
based on new clues, similar to guessing a secret number.
Application in Naive Bayes:
Function: Sorts text into categories like fairy tales, science books, or
adventure stories based on content.
Process: Uses Bayes' theorem to consider how likely words are to appear
in a certain type of text.
Example - Identifying Fairy Tales:
Features: Words in a book like “princess”, “dragon”, and “magic”.
Prior Probability: Initial assumption about the commonness of fairy tales
in the collection.
Calculation: Multiplying probabilities to determine the likelihood of the
book being a fairy tale.
Decision Making: The algorithm compares probabilities across
categories and picks the one with the highest likelihood.
Visualization: Imagine an algorithm weighing words and their
associated probabilities to decide a book's genre.

14/01/2024 9
 Variants of Naive Bayes
Multinomial Naive Bayes:
Approach: Focuses on word frequency rather than mere presence, emphasizing repeated
mentions as indicators.
Application: Effective for textual data, genre classification, spam filtering, and sentiment
analysis.
Visualization: Sorting books with repeated words like “magic”, “princess”, and “dragon” as
amplified clues.
Gaussian Naive Bayes:
Purpose: Tailored for continuous data or data within a specific range.
Strategy: Considers how data points are distributed, visualized as bell-shaped distributions or
“hills.”
Example: Differentiating creatures based on weight distributions, like mice vs. elephants.
Adaptability of Naive Bayes:
Versatility: Suitable for both word counts in text and continuous values in numerical datasets.
Computational Efficiency: Processes vast datasets with ease due to its inherent simplicity.
Overarching Strength:
Bayesian Logic: Guided by probabilistic logic that updates beliefs with incoming evidence,
reflecting dynamic adaptability.
Holistic Approach: Strength lies not just in individual versions but in the overall probabilistic
methodology.

14/01/2024 10
Challenges in Naive Bayes

11
 The Continuous Variable Conundrum
Naive Bayes and Continuous Data: Introduction to the challenges
Naive Bayes faces with continuous variables like height or movie
duration.
Traditional Approach Limitation:
Discrete Focus: Naive Bayes excels in categorical or discrete data but
struggles with continuous ranges.
Gaussian Naive Bayes:
Application: Used for predicting continuous attributes by assuming a
Gaussian distribution.
Visualization: Weights of creatures like mice and elephants represented as
distinct “hills” or bell-shaped curves.
Real-World Complexity:
Challenge: Data often exhibits unpredictable patterns, more like a roller
coaster than smooth hills.
Issue: The assumption of normally distributed data may not always hold,
resulting in inaccuracies.
Implication: A need for adaptable approaches or alternative models
when dealing with non-normal or complex continuous data.

14/01/2024 12
 Knowledge Gaps and Assumptions
Naive Assumption: The belief that each feature or word is
independent of others, simplifying computation but sometimes
missing interdependencies.
Real-World Data Complexity:
Linguistic Nuances: In text classification, the meaning of a word
can be influenced by its neighbors, which Naive Bayes might
overlook.
Dependency on Provided Information:
Prior Knowledge: Uses known information, like common words in
fairy tales, as priors for classification.
Edge Cases Challenge: Struggles with atypical data that doesn't fit
usual patterns, potentially leading to misclassification.
Implications: While Naive Bayes offers computational efficiency,
its simplifications can sometimes lead to inaccuracies,
especially with complex or atypical data.

14/01/2024 13
 The Power of Prior Knowledge
Initial Belief: Naive Bayes starts with a prior probability or initial belief
about the data, influencing initial classifications.
Prior Probability Challenges:
Impact of Inaccurate Priors: Misleading assumptions can lead to
inaccurate predictions if the prior is off or data changes.
Strengths of Naive Bayes:
Simplicity and Scalability: Its greatest asset, making it highly adaptable
and easy to implement.
Versatility: Excels in tasks like email filtering, sentiment analysis, and book
categorization.
Limitations and Adaptability:
Struggle with Continuous Data: May falter when handling continuous
variables or complex dependencies.
Overcoming Pitfalls: Adaptable by refining parameters or incorporating
other techniques to improve accuracy.
Implications: Despite limitations, Naive Bayes' simplicity and scalability
make it a valuable tool in various applications.

14/01/2024 14
Support Vector Machines
(SVM)

15
 Introduction
Support Vector Machines (SVM): A foundational linear classifier in
computational linguistics and machine learning.
High-Dimensional Data Handling: SVM excels at managing and categorizing
high-dimensional datasets, common in text classification.
Kernel Functions: Utilize unique mathematical techniques to transform data
into optimal spaces for classification.
Marble Separation Analogy:
Basic Concept: SVMs aim to separate different types of items (like red and blue
marbles) into distinct categories.
Hyperplane: Finds the "perfect line" or multi-dimensional space (hyperplane) to
separate categories.
Complexity with More Features: As more features (like size and shininess) are
considered, the separation task moves into higher-dimensional spaces.
Objective of SVMs: To find a hyperplane that maximizes the margin between
categories while minimizing classification errors.
SVMs as Helpers: Act as efficient tools for drawing the best line or plane to
differentiate between various elements.

14/01/2024 16
 SVMs and Text Classification
Digital Age Challenge: Managing and making sense of vast amounts of text data
generated every second, from social media to online reviews.
Role of SVM in Text Classification:
Familiar Concept: Similar to the game of “I Spy”, SVMs identify patterns and categories in a
vast expanse of text.
High-Dimensional Navigation: Specializes in high-dimensional environments, finding the
optimal boundary for separating data.
Nature of Text:
Complexity: Every word carries weight, sentiment, and meaning, translating to a multi-
dimensional maze in machine learning.
Example: Differentiating between positive and negative sentiments in online book reviews.
Feature Extraction:
Process: Converting text into a format understandable by algorithms using methods like
Bag of Words or TF-IDF.
Importance: Ensures that words unique to specific categories are emphasized for accurate
classification.
Challenges and Considerations:
Hyperparameters: The performance of SVMs depends on the choice of kernel, cost
parameter, and other settings.
Computational Demand: Handling vast datasets requires efficient preprocessing and
optimization.

14/01/2024 17
 Kernel Functions in SVMs
Kernel Functions in SVMs: Mathematical tools that transform data to better separate categories
in classification tasks.
Role in SVM:
Decision-Making: Kernel functions are crucial when data points are not linearly separable.

Marble Analogy:
Scenario: Separating red and blue marbles scattered on a table without a clear linear division.
Solution: Introducing a third dimension to elevate marbles, creating a multi-level playground for
separation.

Abstract Transformation:
Data Projection: Kernel functions lift data into higher-dimensional space, making it easier to find a
separating boundary.

Types of Kernel Functions:

Linear Kernel: No transformation, attempts direct separation.
Polynomial Kernel: Introduces complex curves to the boundary.
Radial Basis Function (RBF): Focuses on the distance between data points.
Sigmoid Kernel: Adds a two-valued function into the mix.

Strategic Choice:
Alignment with Data: Selecting a kernel function is strategic, aligning its properties with the data's nature
and the task's objectives.
Influence of Data Intricacies: The distribution and specifics of the data guide the choice of the most
effective kernel function.

14/01/2024 18
 Mathematics behind SVMs
Soccer Ball Analogy: Vectors are like superpowered soccer balls flying in many
directions, representing data in the world of SVMs.
SVMs Goal: Sorting data, such as separating dogs from cats in photos or
categorizing messages by sentiment.
Multi-Dimensional Space: The field where vectors (data) exist and interact,
allowing for complex separation tasks.
Separating Boundary: Represented as a super long stick, known as the "normal
vector" or "weight vector", pointing in a specific direction.
Objective: To place the boundary (super-stick) in the optimal position,
maximizing the distance from the closest data points (support vectors).
Training Process: A strategy session involving solving a complex mathematical
puzzle to minimize the norm of the weight vector while ensuring data is
correctly separated.
Support Vectors: The closest data points to the boundary line, critical in
defining the optimal position of the separating line.
Outcome: SVMs effectively separate different types of data in multi-
dimensional spaces using vectors, boundaries, and mathematical optimization.

14/01/2024 19
 SVMs: Dealing with High Dimensional Data
• "I Spy" in a Toy Store Analogy: Each toy type in a gigantic store
represents a dimension, similar to how each unique word is a
dimension in text data.
• High-Dimensional Data: Just as there are many types of toys, text
data has thousands of unique words (dimensions), making
classification complex.
• SVM as a Clever Friend: Skilled at grouping various items (words or
toys) in smart ways, handling high-dimensional data effectively.
• Complex Grouping Challenge: Analogous to grouping toys by
multiple attributes (color, size, and type) simultaneously.
• Kernel Trick: SVM's secret power that simplifies complex data, akin
to magical abilities that highlight specific attributes (like color or size)
to ease classification.
• Adaptability and Skill: Despite the complexity and high
dimensionality, SVMs, with their kernel trick, excel at classification
tasks, similar to a friend winning at a tricky game of “I Spy".

14/01/2024 20
Decision Trees

21
 Decision Trees: A visual and intuitive approach in computational
linguistics for text classification, converting linguistic data into
Introduction discernible patterns.
Mechanics and Building:
Branching Criteria: Understanding the criteria and decisions that
guide the branching of the tree.
Board Game Analogy:
Game Board as Tree: Visualizing the decision tree as a board game
with branches representing decision paths.
Questions as Nodes: Each branching point is a question or rule
guiding the player's path.
Outcomes as Leaf Nodes: End of the branches representing
different outcomes or classifications.
Applicability and Strengths:
Elegance and Simplicity: Ability to elucidate complex textual data
structures with understandable decision paths.
Visual and Intuitive: Easy to understand the decision-making
process, showcasing why certain choices are made.
Practical Implications: Often used when transparency in
decision-making is crucial, allowing users to comprehend the
computer's logic.
14/01/2024 22
 Introduction to Decision Trees: A powerful tool for finding patterns and making
Decision Trees in sense of vast textual data in text classification.

Text Classification Sorting Storybooks Analogy:

Task: Categorizing storybooks into genres like fairy tales, adventure stories, detective
novels, and science fiction.
Function: Decision Trees automate and simplify the classification process, akin to a
wise librarian.
Questioning Strategy:
Method: Uses a systematic series of yes-or-no questions based on specific features
or keywords.
Example: Identifying genres based on the presence of words like “dragon” for fairy
tales or “spaceship” for science fiction.
Transparency and Traceability:
Clarity: Offers a clear roadmap of the decision-making process, allowing easy tracing
of decisions.
Insights: Provides insights into the text data being classified.
Scalability:
Application: Handles vast digital libraries with grace, applicable to online reviews,
news articles, and academic research.
Challenges and Solutions:
Overfitting: The model might become too complex, leading to poor performance on
new data.
Pruning: A technique to trim the tree, ensuring it remains generalized and robust.
14/01/2024 23
 Construction Analogy: Building decision-making models likened to
constructing buildings, requiring tools, methodologies, and expertise.
Building Decision Trees Decision Trees as Treehouses:
Structure: Multi-tiered treehouse where branches represent decisions and leaves
denote outcomes.
Function: Sorts a mountain of storybooks into distinct categories.
ID3 Algorithm:
Foundation: Early design operating with “Information Gain,” determining
questions that bring maximum clarity.
C4.5 Algorithm:
Improvement: Renovated version of ID3, introducing “Gain Ratio” to ensure
balanced splits and avoid chaotic divisions.
CART Algorithm:
Modern Design: Categorizes based on both genre and quantitative features using
“Gini Impurity” for harmonious division.
Choice of Blueprint:
Dependence: Selection based on the specific needs and nuances of the data.
Options: Foundational ID3, balanced C4.5, or versatile CART.
Mastery in Decision Trees:
Architectural Journey: Choosing the right algorithm based on the project's nature
and demands.
Insight and Efficiency: Sculpting a Decision Tree that categorizes data and offers
insights into its logic.

14/01/2024 24
Neural Networks

25
 Introduction to Neural Networks: An advanced paradigm in computational
linguistics inspired by the human brain's interconnected neurons.
Introduction Architecture and Operations:
Anatomy Overview: Understanding the foundational layers and structures of
neural networks.
Text Classification: Tailoring neural networks for various text classification tasks.
Brain and Neuron Analogy:
Human Brain: Billions of neurons working together to understand the world.
Computer Model: Neural networks as computer models mimicking the brain's
learning process.
Learning from Data:
Toy Sorting Analogy: Sorting toys into categories is akin to classifying different
types of text.
Capabilities: Reading and categorizing content like movie reviews (sentiment
analysis) and articles (document classification).
Advancements and Efficacy:
Technological Growth: Enhanced capabilities due to advancements in computing
and data availability.
Performance: Often outperforms older methods in text classification tasks.
Role in Text Classification:
Indispensable Tool: Revolutionized the approach to handling high-dimensional
and intricate textual data.
Popular Choice: Preferred method for various text classification tasks due to
adaptability and computational prowess.

14/01/2024 26
 Anatomy of Neural Neural Networks as Brain Models: Built with interconnected neurons, each
performing mathematical operations to provide outputs.

Networks Structure and Function:

Neuron Functions: Receives input, performs calculations (like adding or
multiplying), and then gives an output.
Layered Organization: Consists of input, hidden, and output layers, each playing a
role in the decision-making process.
Dominoes Analogy:
Process: Lining up dominoes to fall in sequence, akin to the sequential activation
of neurons in different layers.
Decision-Making:
Input Layer: Receives data, such as text from a book.
Output Layer: Makes the final decision, like classifying the book's genre.
Hidden Layers: Assist in refining and guiding the decision process.
Learning Mechanism:
Backpropagation: Identifies which neurons need adjustment for better decision-
making.
Gradient Descent: Determines the extent of adjustments to improve accuracy.
Continuous Improvement: With each adjustment, the neural network becomes
more proficient.
Importance of Learning: Essential for neural networks to learn from data and
make increasingly accurate decisions.

14/01/2024 27
 Types of Neural Networks Setting the Scene: A grand theater where a magician reveals neural
networks as mystical boxes turning words into stories.
for Text Classification Three Types of Neural Networks:
Feedforward Neural Networks: One-directional journey of words, akin to a
magical tunnel transforming inputs into a definite story type.
Convolutional Neural Networks (CNNs): Detective-like, seeking patterns
within text with a magical magnifying lens, identifying recurring themes or
phrases.
Recurrent Neural Networks (RNNs): Box with a memory, linking past with
present through interconnected loops, understanding the full context of
the story.
Magician’s Unveiling:
Feedforward Network: Straightforward processing, words go in, get
refined, and emerge as a story.
CNN: Zooms into sequences, deduces narrative style by recognizing textual
patterns.
RNN: Remembers past text parts, ensuring the story is understood in full
context.
Demonstration Outcome:
Unique Mechanisms: Each network offers a distinct approach to
understanding and classifying text.
Masterpieces of Design: Not mere tricks but brilliant designs in machine
learning, crafting jumbled words into discernible narratives.

14/01/2024 28