Advanced Machine Learning and Deep Learning

Models for Classifying Bengali Toxic Comments

Miftahul Sheikh Adibul Haque

Computer Science and Engineering Computer Science and Engineering
Ahsanullah University of Science and Technology Ahsanullah University of Science and Technology
Dhaka, Bangladesh Dhaka, Bangladesh
[email protected] [email protected]

Md. Yousuf Ali Tahsin Shadman

Computer Science and Engineering Computer Science and Engineering
Ahsanullah University of Science and Technology Ahsanullah University of Science and Technology
Dhaka, Bangladesh Dhaka, Bangladesh
[email protected] [email protected]

Abstract—Social media is now widespread, making online com- Bullying, whether physical, verbal, emotional, or social, can
munication easy. Users can share information, collaborate, and cause immediate or long-term harm. According to the CDC,
interact positively. However, social media can also become toxic, bullying involves repeated, unwelcome hostile behavior with
leading to online abuse and bullying, particularly affecting young
people and celebrities, causing emotional and psychological harm. an imbalance of power. With the rise of digital technology,
To address this, we use Natural Language Processing (NLP) cyberbullying has become a significant issue. Cyberbullying
and Machine Learning (ML) to detect and prevent toxic and happens online through smartphones, blogs, and social media
non-toxic comments. Algorithms such as Perceptron, K-Nearest platforms like Facebook, YouTube, and Twitter. It can involve
Neighbour (K-NN), and Logistic Regression (LR), alongside Deep sending threatening messages, posting abusive comments, or
Learning models like Long Short-Term Memory (LSTM) and
Convolutional Neural Networks (CNN), as well as Transformer- spreading rumors, often anonymously, making it hard to con-
based models like BERT, help identify abusive language. This trol.
paper presents a deep learning-based pipeline for categorizing In Bangladesh, increasing internet use has led to more
Bengali toxic comments. First, a binary classification model online harassment, especially against women. A survey by
determines whether a comment is toxic or not. If toxic, another ActionAid Bangladesh found that 50% of women experienced
model categorizes the comment into one or more of six toxic
categories: vulgar, hate, religious, threat, troll, and insult. Our online harassment, with over 62% of victims under 25. Face-
manually labeled dataset consists of 16,073 instances, with 8,488 book was identified as the main platform for harassment,
labeled as toxic, collected from https://github.com/deepu099cse/. causing mental health issues for many women.
We performed text preprocessing and applied seven feature Researchers are using artificial intelligence (AI) to create
extraction techniques. We tested three ML algorithms, two DL systems that detect offensive content on social media. Many
models, and a pre-trained Transformer model (BERT) to evaluate
their performance. Our findings show that BERT achieved the studies focus on Bengali, one of the most spoken languages
highest accuracy at 90.01%, outperforming other algorithms. worldwide. However, better methods are needed to accurately
Keywords— Toxic comment, NLP, Machine learning, find and remove toxic Bengali content.
Deep learning, Transformer model, BERT. Our study aims to improve the detection of toxic and non-
toxic comments in Bengali using machine learning (ML) and
I. I NTRODUCTION deep learning (DL) techniques, focusing on binary classifi-
The rapid growth of social media and mass media has cation. We explored two ML methods (Logistic Regression
made it easier for people to share their thoughts and emotions. and Random Forest), two DL methods (LSTM and CNN),
However, this also means harmful content can spread quickly. and a transformer-based model (Bangla-bert-base). We used
Toxic content includes anything that makes someone feel seven word embedding techniques, including TF-IDF, Bag of
uncomfortable or hurt, such as sexually explicit material, Words, Word2Vec, GloVe, and FastText. Our results showed
violent content, hate speech, cyberbullying, and harassment. that the Random Forest model with FastText performed best
This type of content often attracts more attention online. among feature extraction-based models, and the LSTM model
outperformed other ML models. Overall, the Bangla-bert-base
model gave the best results.
 II. LITERATURE REVIEW toxic behaviors. The distribution of toxic comments is as
Numerous studies have explored the use of machine learning follows: vulgar (15.59%), hate (11.81%), religious (8.82%),
techniques to detect cyberbullying. Here, we highlight some threat (8.83%), troll (10.22%), and insult (16.92%), with insult
significant works conducted in various languages. and vulgar comments being the most prevalent.
A paper [1] presents a deep learning-based pipeline for cat- The text samples were gathered from the Bangla-Abusive-
egorizing Bengali toxic comments. Using LSTM with BERT Comment-Dataset, Bengali Hate Speech Dataset, and Bangla
Embedding, the model achieved 89.42% accuracy in binary Online Comments Dataset. Due to inconsistencies and inaccu-
classification. For multi-label classification, CNN-BiLSTM racies in the original labeling, texts were manually reclassified
with attention mechanism achieved 78.92% accuracy and 0.86 into six categories: vulgar, hate, religious, threat, troll, and
weighted F1-score. insult. This reclassification ensures consistency, enhances data
A paper [2] discusses the use of CNN and LSTM models quality, and improves the accuracy of machine-learning models
for detecting toxic Bengali comments on social media. CNN by providing a clearer understanding of the toxicity present in
achieved the highest accuracy of 95.30%, outperforming Naive the data.
Bayes (81.80%), SVM (84.73%), and Logistic Regression
(85.22%). Both deep learning models surpassed traditional
approaches by a 10% margin.
A paper [3] systematically examines online harassment and
classifies toxic comments using six machine learning algo-
rithms. Logistic Regression achieves the highest accuracy at
89.46% and the lowest Hamming loss at 2.43%, outperforming
Naive Bayes, Decision Tree, Random Forest, KNN, and SVM.
A paper [4] discusses the use of the L-Boost algorithm,
combining AdaBoost and LSTM with BERT, to detect of-
fensive Bengali and Banglish texts on social media. The
study achieved 95.11% accuracy with L-Boost, outperform-
ing Bangla BERT at 88.47%, LSTM-BERT at 89.45%, and
AdaBoost-BERT at 92.16%.
Fig. 1. Distribution of Classes
A paper [5] focuses on detecting hate speech and cyber-
bullying in Bangla comments from social media. It uses a
dataset of 2000 comments and evaluates classifiers like GRU, B. Text preprocessing
Logistic Regression, Random Forest, MNB, and SVM. The In data analysis, making data understandable for machines
study aims to differentiate anti-social from socially acceptable is crucial. Before applying machine learning (ML) and deep
comments and address the lack of Bangla-specific anti-social learning (DL) algorithms, we need to process the data through
classification research. filtering and tokenization. Normal texts often contain symbols
A paper [6] addresses the issue of harmful online comments and words from various languages, so filtering is necessary
by utilizing a hybrid model combining Convolutional Neural to clean the data. In our research, we removed digits, web-
Networks (CNN) and Long Short-Term Memory (LSTM) site links, punctuation, symbols, signs, emoticons, and any
networks. The approach pre-processes text data and uses word characters that were not Bengali. We also eliminated Bengali
embeddings for classification, achieving notable performance stopwords. Tokenization involves breaking down the text into
metrics like accuracy, precision, recall, and F1-score. smaller pieces, called tokens, using spaces as delimiters. After
III. METHODOLOGY this, we applied lemmatization, which reduces words to their
base or root form, helping to standardize the dataset. Figure
The study is divided into five parts. First, we collect and 3 shows a sample from our dataset after this filtering process.
prepare the dataset. Next, we preprocess the data. Then, we
extract features from the data. After that, we apply machine C. Conversion to binary class from multi-class
learning algorithms. Finally, we evaluate the performance of For our experiment, we converted our dataset from a multi-
these algorithms. Figure 5 shows the entire proposed method- class format to a binary classification format. We categorized
ology in a block diagram. troll, sexual, religious, and threat texts as bullying (represented
as 1), while neutral comments were labeled as non-bullying
A. Dataset collection and preparation
(represented as 0). This conversion resulted in a dataset with
The dataset utilized in this study comprises 16,073 user 28,661 bullying texts (1) and 15,340 non-bullying texts (0).
comments from social media platforms, focusing on Bangla The binary class distribution can be seen in Figure 4.
cricket-related content. It includes eight variables: SL No.,
Text, vulgar, hate, religious, threat, troll, and insult. The D. Feature extraction
Text column contains the actual comments, while the other Feature extraction is essential for training machine learn-
columns indicate the presence (1) or absence (0) of various ing models with text data, as it converts words into nu-
 X X
J(θ) = log P (c | w) (4)
w∈corpus c∈context(w)

exp(vc · vw )
P (c | w) = PV (5)
i=1 exp(vi · vw )

where:
• J(θ) is the objective function to be maximized.
• w is the target word.
• c is a context word.
• vc and vw are the vector representations of the context
word and the target word, respectively.
• V is the vocabulary size.
Fig. 2. Binary Class Distribution
4) GloVe: GloVe is a word embedding technique that
captures global word co-occurrence statistics from a corpus.
merical representations. In our study, we used seven word Unlike Word2Vec, which focuses on local context, GloVe
embedding techniques: Term Frequency-Inverse Document constructs word vectors by utilizing the overall frequency of
Frequency (TF-IDF), Bag of Words (Uni-gram, Bi-gram, Tri- word pairs throughout the entire text. This method effectively
gram), Word2Vec, GloVe, and FastText. These methods allow encodes semantic relationships and analogies between words.
us to handle text data effectively for analysis.
1) TF-IDF: Term frequency-Inverse document frequency V
X  2
vectorizer (TF-IDF) is a vectorization technique that converts J= f (Xij ) wiT w̃j + bi + b̃j − log Xij (6)
text data into vectors. From raw texts, it creates a matrix of i,j=1

features. where:
• J is the cost function to be minimized.
TF-IDF(t, d, D) = TF(t, d) × IDF(t, D) (1) • V is the vocabulary size.

Here, t is the term and d is a document. D is the entire • Xij is the co-occurrence count of word i and word j.

corpus of documents. • wi and w̃j are the word vectors for word i and word j,

Term Frequency (TF): TF counts the number of times a term respectively.

appears in a document. TF is calculated using the equation 2. • bi and b̃j are the biases for word i and word j, respec-
tively.
• f (Xij ) is a weighting function that assigns less weight
Number of times term t appears in document d
TF = (2) to very frequent and very infrequent co-occurrences.
Total number of terms in document d
5) FastText: FastText represents words using character n-
Inverse Document Frequency (IDF): This is a weight that is
grams, with each word’s embedding being the sum of its
an indicator of how frequently a term is used in a document.
character n-gram vectors. This enables FastText to create
Its score decreases with increased usage across a document
embeddings for both known words and unseen words that
and scales up the less frequent words using equation 3.
share similar character n-grams with the trained words.
 
Total number of documents in corpus w=
X
zg (7)
IDF = log +1
Number of documents containing term g∈G(w)
(3)
2) Bag of words: A uni-gram represents individual words where:
in a text corpus, treating each word independently without • w is the vector representation of the word.
considering the words around it. In contrast, a bi-gram involves • G(w) is the set of character n-grams in the word w.
pairs of consecutive words, which allows it to capture some • zg is the vector representation of the character n-gram g.
contextual information from neighboring words. Going a step
further, a tri-gram considers sequences of three consecutive E. Bangla-Bert
words, thereby providing even more contextual insight com- Bidirectional Encoder Representations from Transformers,
pared to bi-grams. or BERT, is a deep learning model built on the Transformer
3) Word2Vec: Word2Vec is a widely used NLP technique architecture. Unlike traditional models that process text in a
for creating word embeddings, where words are represented single direction, BERT reads text in both directions simulta-
as numerical vectors in a continuous space. It learns these neously, allowing it to better capture context. In BERT, each
embeddings by analyzing the context in which words appear output element is connected to every input, with dynamic
within a text corpus. attention-based weighting determining the importance of each
element. This makes BERT not just an embedding technique, 2) Precision: Precision assesses the ratio of true positive
but a comprehensive language model. predictions (correctly identified positive instances) to the total
For Bengali, the pre-trained model ”bangla-bert-base” uti- number of positive predictions made. It emphasizes the accu-
lizes masked language modeling, a technique detailed in racy of positive predictions.
BERT. This model was trained using two primary sources:
TP
the Open Super-large Crawled Aggregated coRpus (OSCAR) Precision = (9)
and Bengali Wikipedia. It follows Google BERT’s architec- TP + FP
ture, featuring 12 layers, 768 hidden units, and 110 million 3) Recall: This metric calculates the ratio of true positive
parameters. predictions to all actual positive instances. It focuses on the
To implement BERT, we used the ”simple transformer” NLP model’s ability to identify all relevant positive cases.
library and configured it with a ”Classification Model” for
TP
binary classification tasks. Recall = (10)
TP + FN
4) F1-score: The F-measure is a single metric that com-
bines precision and recall using their harmonic mean. When
precision and recall are given equal importance, this metric is
referred to as the F1-score.
2 · Precision · Recall
F1-score = (11)
Precision + Recall
IV. RESULT ANALYSIS
We assessed the performance of our toxic and non-toxic
comments detection models using essential metrics including
accuracy, precision, recall, and F1-score.
Fig. 3. Proposed methodology for toxic comment classification.

F. Classification
We tested three machine learning classifiers (K-NN, Lo-
gistic Regression, and Random Forest), two deep learning
classifiers (LSTM and CNN), and one pre-trained BERT
model, ”bangla-bert-base,” to train our dataset and classify
Bengali cyberbullying texts.
Subsequently, we evaluated and compared their performance
using various performance metrics.

G. Performance metrics
Evaluation metrics are essential in machine learning for
assessing a model’s performance. They provide a way to
measure how accurately a model’s predictions match the actual
values, helping to quantify its effectiveness.
1) Accuracy: This metric evaluates the proportion of cor- Fig. 4. Result Analysis Table
rect predictions out of the total number of predictions made.
It is particularly useful when the classes in the dataset are Our study revealed that Bangla-BERT outperformed all
balanced. other algorithms, achieving an accuracy of 90.01%. Linear
Regression (LR) model, when combined with the GloVe em-
TP + TN bedding technique, also performed well, achieving an accuracy
Accuracy = (8)
TP + TN + FP + FN of 82%. This model’s precision is 0.82, recall is 0.86, and F1-
score is 0.84.
Here: Among deep learning models, both LSTM and CNN per-
• T P = True Positives formed very well. LSTM achieved the highest accuracy among
• T N = True Negatives deep learning models with an accuracy of 89%. CNN achieved
• F P = False Positives a good accuracy of 88%. But, we found that BI-gram and TRI-
• F N = False Negatives gram feature extraction methods did not perform as well as
 [3] Kajla, Harsh, Jatin Hooda, and Gajanand Saini. ”Classification of
online toxic comments using machine learning algorithms.” In 2020 4th
international conference on intelligent computing and control systems
(ICICCS), pp. 1119-1123. IEEE, 2020.
[4] Sarker, Manash, Md Forhad Hossain, Fahmida Rahman Liza, Syed
Nazmus Sakib, and Abdullah Al Farooq. ”A machine learning approach
to classify anti-social Bengali comments on social media.” In 2022
International conference on advancement in electrical and electronic
engineering (ICAEEE), pp. 1-6. IEEE, 2022.
[5] Mridha, Muhammad Firoz, Md Anwar Hussen Wadud, Md Ab-
dul Hamid, Muhammad Mostafa Monowar, Mohammad Abdullah-Al-
Wadud, and Atif Alamri. ”L-boost: Identifying offensive texts from
social media post in bengali.” Ieee Access 9 (2021): 164681-164699.
[6] Bhattacharya, Sampurna, Bhavani Gowri Shankar, B. Pitchaimanickam,
and A. Alice Nithya. ”Toxic Comments Classification using LSTM
and CNN.” In 2024 3rd International Conference on Applied Artificial
Intelligence and Computing (ICAAIC), pp. 214-221. IEEE, 2024.

Fig. 5. Accuracy Graph (%)

Fig. 6. Classifiers Result Graph

other techniques. Both BI-gram and TRI-gram models yielded

similar results, which were lower compared to the perfor-
mances of more advanced methods like word2Vec, GloVe, and
FastText embedding techniques.
V. CONCLUSION AND FUTURE WORKS
The study highlights the effectiveness of advanced mod-
els, particularly Bangla-BERT, in improving the detection of
Bengali toxic comment. Bangla-BERT achieved the highest
accuracy of 90.01%, outperforming other models such as the
Logistic Regression (LR) with GloVe embeddings and the
Long Short-Term Memory (LSTM) model. Future research
should aim to expand the dataset to encompass a broader range
of texts, investigate additional state-of-the-art transformer-
based models, develop real-time detection systems for social
media platforms, integrate multilingual capabilities, and incor-
porate user feedback mechanisms to continually enhance the
model’s accuracy and relevance.
R EFERENCES
[1] Belal, Tanveer Ahmed, G. M. Shahariar, and Md Hasanul Kabir. ”Inter-
pretable multi labeled bengali toxic comments classification using deep
learning.” In 2023 International Conference on Electrical, Computer and
Communication Engineering (ECCE), pp. 1-6. IEEE, 2023.
[2] Banik, Nayan, and Md Hasan Hafizur Rahman. ”Toxicity detection on
bengali social media comments using supervised models.” In 2019 2nd
international conference on Innovation in Engineering and Technology
(ICIET), pp. 1-5. IEEE, 2019.