Int J Syst Assur Eng Manag

https://doi.org/10.1007/s13198-024-02597-0

ORIGINAL ARTICLE

Emotion detection in text: advances in sentiment analysis

R. Tamilkodi1 · B. Sujatha1,2 · N. Leelavathy1,2

Received: 24 September 2024 / Revised: 16 October 2024 / Accepted: 24 October 2024

© The Author(s) under exclusive licence to The Society for Reliability Engineering, Quality and Operations Management (SREQOM), India and
The Division of Operation and Maintenance, Lulea University of Technology, Sweden 2024

Abstract In the era of rapid internet expansion, social methods. For 50 clients, TBM accomplished an precision
networking platforms have become indispensable chan- of 94.23%, accuracy of 89.13%, and review of 91.59%. As
nels for individuals to convey their emotions and opinions the client check expanded to 100, 150, 200, and 250, TBM
to a global audience. People employ various media types, reliably outflanked others, coming to up to 97.03% precision,
including text, images, audio, and video, to articulate their 92.89% exactness, and 93.51% review. These comes about
sentiments. However, the sheer volume of textual content emphasize the viability of the TBM approach over elective
on web-based social media platforms can be overwhelm- strategies.
ing. These platforms generate an enormous amount of
unstructured data every second. To gain insights into human Keywords Sentiment analysis · Text detection ·
psychology, it is imperative to process this data as quickly Identifying emotions · Positive · Negative
as it is produced. This can be achieved through sentiment
analysis, an advanced technique called Transformer-based
model (TBM) which discerns the polarity of text, determin- 1 Introduction
ing whether the author holds a positive, negative, or neutral
stance towards a subject, service, individual, or location. The Sentiment analysis, which is alternatively referred to as opin-
performance of this model can vary based on factors like the ion mining, is a facet of natural language processing (NLP)
dataset used, the specific Transformer variant, model hyper that revolves around the assessment of the sentiment or emo-
parameters, and the evaluation metrics employed. Findings tional undertone conveyed within a text. This proficient tool
from this study show that social media users with depression is instrumental in deciphering public sentiment, evaluating
or anorexia may be identified by the presence and unpre- customer feedback, and gauging the sentiment prevailing on
dictability of their emotions. The proposed model is used social media regarding various subjects, products, or ser-
to analyze text data and make sentiment predictions effec- vices. Transformer-Based Sentiment Analysis Models are a
tively. The proposed TBM strategy illustrated predominant cutting-edge approach to tackle this task. Sentiment analysis
execution over distinctive measurements compared to other aims to classify text into different sentiment categories, typi-
cally "positive," "negative," or "neutral." More fine-grained
* R. Tamilkodi approaches might include additional categories like "neu-
[email protected] tral," "slightly positive," "very negative," and so on. Senti-
B. Sujatha ment analysis models based on the Transformer architecture
[email protected] leverage the groundwork established by Vaswani et al. in
N. Leelavathy (2017a, b). Transformers represent neural network designs
[email protected] crafted for effective processing of sequential data, rendering
1
Department of CSE (AIML&CS), Godavari Global them particularly apt for tasks within the domain of natural
University, Rajahmundry, A.P, India language processing (NLP).
2
Department of CSE, Godavari Global University, At the heart of Transformers lies the pivotal innovation of
Rajahmundry, A.P, India the self-attention mechanism, enabling the model to assess

Vol.:(0123456789)
 Int J Syst Assur Eng Manag

the significance of individual words within a sentence. This improvements in sentiment analysis using Transformer-
capability empowers the model to capture intricate relation- based architectures. "Attention Is All You Need" (Vaswani
ships and interdependencies effectively. To adapt pre-trained et al., 2017a, b): This is the seminal paper introducing the
Transformers for sentiment analysis, researchers fine-tune Transformer architecture. While not focused solely on senti-
these models on sentiment-labeled datasets. During fine-tun- ment analysis, it laid the foundation for using Transformers
ing, the models learn to map input text to sentiment labels. in various NLP tasks, including sentiment analysis.
In the fine-tuning process, a common step entails incor- "BERT: Bidirectional Encoder Representations from
porating a classification head into the pre-trained model. Transformers" (Devlin et al., 2018a, b): This paper intro-
This classification head utilizes the encoded representation duces BERT, a pre-trained Transformer model that set new
of the input text to make predictions regarding sentiment benchmarks for performance across a diverse array of NLP
labels, such as positive, negative, or neutral. Transformer- tasks, including sentiment analysis. BERT’s capacity to
Based Sentiment Analysis Models benefit from transfer capture contextual information ushered in a transformative
learning. The pre-trained models have already learned a era in sentiment analysis."XLNet: Generalized Autoregres-
broad understanding of language, syntax, and semantics sive Pretraining for Language Understanding" (Yang et al.,
from massive datasets, making them effective at capturing 2019): XLNet, another pre-trained Transformer model,
sentiment nuances. Fine-tuning requires less labeled data advanced the state of the art in NLP tasks. Researchers have
compared to training from scratch. Sentiment analysis can explored its efficacy in sentiment analysis with remarkable
be challenging due to context, sarcasm, and cultural nuances outcomes.
in language. While Transformer-based models have signifi- "RoBERTa: A Robustly Optimized BERT Pretraining
cantly improved accuracy, they are not immune to bias and Approach" (Liu et al., 2021): RoBERTa, a variant of BERT,
limitations. enhances performance by fine-tuning hyper parameters and
Ongoing research focuses on addressing these chal- training data, resulting in notable improvements across vari-
lenges, including reducing bias, handling multilingual sen- ous NLP tasks, including sentiment analysis."ELECTRA:
timent analysis, and dealing with noisy or short text data Pre-training Text Encoders as Discriminators rather than
from social media. Transformer-Based Sentiment Analysis Generators" (Clark et al., 2020): ELECTRA, an innovative
Models have found applications in various industries, such pre-training approach, has been applied to sentiment analysis
as marketing, e-commerce, finance, and healthcare, where tasks. It introduces a novel training paradigm for language
understanding customer sentiments and opinions is crucial models that can prove advantageous in the context of senti-
for decision-making. ment analysis when fine-tuned.
The paper presents a novel Transformer-Based Demon- "GPT-3: Language Models are Few-Shot Learners"
strate (TBM) for estimation examination that essentially (Brown et al., 2020): While GPT-3 primarily focuses on text
upgrades the exactness of identifying feelings in social generation, it has also shown promise in sentiment analy-
media content. The oddity lies in its utilize of self-attention sis. Researchers have explored its capabilities for sentiment
instruments and fine-tuning on sentiment-labeled datasets classification tasks."Unified Language Model Pre-training
to capture relevant data viably. The proposed demonstrate for Natural Language Understanding and Generation" (Liu
outflanks existing approaches with up to 97.03% exactness, et al., 2021): The paper introduces GPT-4, a successor to
92.89% exactness, and 93.51% review. Its capacity to recog- GPT-3. While GPT-4 continues to excel in text generation,
nize clients with conditions like discouragement or anorexia it’s expected to be relevant in sentiment analysis due to its
based on enthusiastic eccentrics highlights its commonsense advanced language understanding capabilities.
application in mental wellbeing observing and real-time "Fine-Tuning Transformers for Sentiment Analysis" (Yin
assumption expectation. These enhancements make it well- et al., 2020): This study delves into the fine-tuning process
suited for real-time usage in different businesses. Aragón for Transformer models in sentiment analysis tasks, high-
et al. (2019), Chalkidis et al. (2020), Chikersal et al. (2020), lighting best practices for optimizing model performance on
Dai et al. (2019),Ding et al. (2021), Dong et al. (2021), Gil- various sentiment datasets. "Multi-Modal Sentiment Analy-
lioz et al. (2020), Gururangan et al. (2020a), Gururangan sis using Transformer Models" (Biswas et al., 2020): Some
et al. (2020b), Jia et al. (2021). research extends Transformer-based models to perform sen-
timent analysis on multi-modal data, including text, images,
and audio. This paper explores such multi-modal sentiment
2 Literature review analysis using Transformers.
"Sentiment Analysis in Social Media with Pre-trained
A literature review of Transformer-Based Sentiment Analy- Language Models" (Xia et al., 2021): This work discusses
sis Models would typically encompass a range of papers the application of pre-trained language models, specifically
and studies that explore the development, applications, and Transformers, in sentiment analysis of social media content,
Int J Syst Assur Eng Manag

where language can be informal and nuanced. This literature

review highlights some of the key papers and research direc- Input
tions in the field of Transformer-Based Sentiment Analysis
Models. Researchers have continued to explore and inno-
vate in this area, making significant advancements in sen- Data Collecon
timent analysis using Transformer architectures Jin et al.
(2024), Khanuja et al. (2019), Li et al. (2019), Rahali and
Akhloufi (2023), Ramirez-Cifuentes and Freire (2018), Sun
and Dehghani (2019), Sun et al. (2019), Xiao et al. (2019), Data Preprocessing
Xuetong et al. (2018).

Feature Extraction
3 Proposed system TBM architecture

We proposed a novel transformer architecture that incorpo-

rates whether the person holds a positive, negative, or neu- Emotion Classification
tral instance towards a subject, service, individual, or loca-
tion (Fig. 1). The model is pre-trained on a large corpus of
text data and fine-tuned on a diverse set of downstream tasks Analyze Results
which includes typically "positive," "negative," or "neutral."
More fine-grained approaches might include additional cat-
egories like "neutral," "slightly positive," "very negative,"
Output
and so on. Sentiment classification (e.g., classifying product
reviews as positive, negative, or neutral). Aspect-based sen-
timent analysis which identifies sentiments towards specific
Fig. 2  Sentiment analysis process using transformer-based model
aspects in text, e.g., product features. Emotion detection by
detecting specific emotions expressed in text. To enhance
the model’s performance. These models leverage attention followed by data preprocessing to clean and format the
mechanisms to capture contextual information effectively, information. Feature extraction is then performed using
allowing them to understand the nuances of language and tools like General Inquirer and Sentiwordnet to capture
context in sentiment-bearing text. sentiment-related features. The core step involves emo-
The flowchart shown in Fig. 2 outlines the senti- tion classification using the transformer-based model, cat-
ment analysis process using a transformer-based model, egorizing sentiments as positive, negative, or neutral. The
as discussed in the paper. It begins with data collection results are then analyzed for performance metrics such as
from various sources like Amazon and Rotten Tomatoes, accuracy and precision. Finally, the findings are outputted
for evaluation, concluding the process.

Fig. 1  Architecture of the

proposed system TBM Input Text Tokenizaon Embedding Layer

Positional Encoding

Multi-head self -Attention

Feed-Forward Neural Layer Normalization Residual Connections

Networks

Output layer
 Int J Syst Assur Eng Manag

4 Working principle of TBM 5 Proposed model TBM algorithm

First step involved in this model is to input the text (e.g., a The following are the steps involved in the proposed model.
sentence or document) which is tokenized into smaller units, Step 1: Data preprocessing
typically words or sub word tokens (e.g., using Byte-Pair
Encoding or Word Piece).Each token is represented as an • Load and preprocess the dataset. Tokenize the text
embedding vector. These embeddings capture the seman- data into sub word units (e.g., using BERT Word Piece
tic meaning of the words or sub words in the input. Trans- tokenizer).
formers do not have built-in information about the order • Convert text tokens into numerical representations (e.g.,
of words in a sequence. To address this, positional encod- using word embeddings like Word2Vec, GloVe, or pre-
ings are incorporated into the word embeddings, providing trained Transformer embeddings like BERT embed-
the model with positional awareness for each token within dings).
the sequence. The pivotal breakthrough of the Transformer • Prepare the data by creating input sequences, adding
architecture lies in its self-attention mechanism, enabling padding, and generating corresponding labels (sentiment
the model to assess the significance of each word or token labels).
relative to others in the sequence. This self-attention com-
putation involves considering all other tokens in the input Step 2: Customize the final layers for sentiment analysis
sequence for each token, facilitating the model’s adept cap- using anyone of the architecture Multi-head self-attention
ture of contextual information. layers, Positional encodings to capture the order of words,
The incorporation of multi-head functionality implies that Feed-forward neural networks, Layer normalization and
self-attention operates in parallel multiple times, enabling residual connections. Add a classification head on top of
the model to concurrently emphasize various facets of the the Transformer encoder layer which typically includes a
input. Following self-attention, the output undergoes addi- softmax layer to predict sentiment labels (e.g., positive,
tional refinement via a position-wise feed forward neural negative, neutral).
network. This network independently applies a linear trans- The TBM follows the encoder-decoder structure typical
formation, followed by a non-linear activation function (such of transformer models, where the output of the encoder feeds
as ReLU), to each position or token within the sequence. into the decoder. The key components of the transformer
Residual connections (skip connections) are employed to architecture are
help mitigate the vanishing gradient problem in deep net-
works. These connections allow gradient information to flow 5.1 Self‑attention mechanism
easily through the network. Layer normalization is applied to
the output of each sub-layer (self-attention and feed forward) This mechanism computes the attention score A for a given
to stabilize training. input sequence X as follows
The core Transformer model consists of multiple stacked � √ �
Transformer layers (encoder layers). Each layer refines the A = softmax QK T � dk V
representation of the input. In sentiment analysis, you might
use pooling techniques like max-pooling or mean-pooling where Q is the query matrix, K is the key matrix, V is the
to summarize the representations of all tokens into a fixed- value matrix, dk is the dimension of the key vectors
length vector. The final layer(s) of the model produce the
sentiment prediction. In binary sentiment analysis, you
5.2 Loss function
typically have a single output neuron with a sigmoid activa-
tion function that predicts the sentiment score (positive or
The loss function used for training the TBM can be
negative). In multi-class sentiment analysis, you may have
expressed using the categorical cross-entropy loss, which
multiple output neurons, one for each class (e.g., positive,
is suitable for multi-class classification tasks in sentiment
negative, neutral).The model is trained using labeled data,
analysis. It is defined as:
where the sentiment labels are used as the target. Typical
loss functions encompass binary cross-entropy for tasks ( ) C
∑
involving binary sentiment analysis and categorical cross- L y, y∧ = − yi log(y∧i )
entropy for those involving multi-class sentiment analysis. i=1

Gradient descent algorithms (e.g., Adam) are used to opti- where Y is the true lable (one-hot encoded), y∧ is the pre-
mize the model’s parameters. During inference, the trained dicted probability distribution, C is the number of classes
model takes an input text, processes it through the layers, (e.g. positive, negative, and neutral).
and produces a sentiment prediction.
Int J Syst Assur Eng Manag

Step 3: Training Table 1  Number of reviews taken from the benchmark dataset
Topic Positive Negative Neutral Total tweets
• Initialize model weights randomly or with pre-trained tweets tweets tweets
Transformer weights (transfer learning).
• Define a loss function suitable for sentiment analysis Google 209 602 87 898
Twitter 91 439 238 768
(e.g., cross-entropy loss).
• Define hyper parameters, including learning rate, batch Microsoft 106 678 345 1129
Facebook 185 589 231 1005
size, and the number of training epochs.
• Partition your dataset into training, validation, and test Instagram 79 424 145 648

subsets.
• Commence model training by back propagation and gra-
dient descent (e.g., Adam optimizer). Monitor perfor- others. The dataset includes surveys from seven distinctive
mance on the validation set to prevent over fitting. spaces, with the Blitzer et al. (2006) dataset utilized for
assessing cross-domain assumption classification. This dif-
Step 4: Evaluate the test dataset’s performance using fering dataset incorporates between 800 and 1,000 labeled
metrics like accuracy, precision, recall, F1-score, and the and unlabeled audits per space, giving a vigorous establish-
confusion matrix. ment for preparing and testing the model. Reviews were
Step 5: Fine-tuning collected from general websites like amazon.com, rotten
tomatoes, televisions.reviewed.com, digitaltrends.com, dvd-
• If the model’s performance is not satisfactory use fine- talk.com, pcmag.com, gadgets.ndtv.com, bestproducts.com,
tune hyper parameters or experiment with different americastestkitchen.com, cooksillustrated.com, cnet.com
Transformer variants. and reviews.com for 7 different domains. Also, the dataset of
• Consider using techniques like early stopping to prevent Blitzer et al. (2006) is used in evaluating the work on cross-
over fitting. domain sentiment classification. The dataset contains both
labelled and unlabelled reviews ranging from 800 to 1,000
Step 6: Inference by using the trained model for sentiment reviews for each domain. The implementation has been done
analysis on new, unseen text data. in Java. For POS tagging, Standford Parser is used. General
Inquirer, Sentiwordnet and Wordnet standard dictionaries
have been used in feature extraction and constructing polar-
6 Results and discussion ity lexicon. Table 2 shows the total number of reviews of
different domains analyzed.
Our experimental results demonstrated that the proposed The benchmark dataset in Table 1 was used for train-
transformer-based model outperforms existing models ing. Threefold cross-validation is applied for carrying out
on a range of benchmark datasets. It achieves significant the experiments. The data was randomly divided into three
improvements in accuracy, achieving state-of-the-art results folds, each containing 200 positive, 200 negative, and 200
on Amazon product Reviews, Yelp Reviews, Twitter Sen- neutral documents. The experiment involved incrementally
timent Analysis Datasets, Customer review datasets and increasing the number of training samples.
product review datasets etc. Moreover, our model exhibits In order to perform result analysis of the proposed TBM,
improved efficiency in terms of training time and computa- standard twitter application programming interface (API)
tional resources. Our exploratory comes about demonstrate was developed. The API fetched real-time tweets for 125
that the proposed transformer-based show (TBM) altogether test users which had participated in this evaluation. Videos
outflanks existing models over different benchmark datasets, were recorded for each of these users during the evaluation
accomplishing state-of-the-art exactness on Amazon item process, where they were asked about their thoughts and
audits, Cry surveys, Twitter estimation examination data- emotions while posting the tweets. Facial reactions of users
sets, client audit datasets, and other item survey datasets. were recorded during this evaluation, and given to the pro-
The progressed precision illustrates the model’s viability and posed model for analysis. Parametric evaluation was done
flexibility in opinion examination applications. using accuracy, precision, recall and F1-Score parameters,
and compared with other works.
6.1 Dataset collection
6.2 Training and assessment methodology
As in Table 1, a comprehensive dataset was collected from
different sources, counting well-known stages such as ama- To approve the execution of the proposed TBM, we
zon.com, spoiled tomatoes, and digitaltrends.com, among utilized triple cross-validation, as sketched out in the
 Int J Syst Assur Eng Manag

Table 2  Precision for proposed No.of users Precision Precision Precision Precision Precision Precision
TBM model with different
works Vaswani Ramponi Trotzek et al. (2018) Kiesel (2019) Yin (2020) Proposed TBM
et al. et al.
(2017b) (2020)

50 73.13 76.45 78.9 83.94 87.72 89.13

100 76.84 79.81 78.3 80.73 89 91.12
150 74.04 77.58 79.26 81.27 88.27 91.76
200 75.73 78.96 80.18 80.22 88.57 92.12
250 74.67 77.92 79.65 80.89 88.49 92.89

Table 3  Recall for proposed No.of users Recall Recall Recall Recall Recall Recall
TBM model with different
works Vaswani Ramponi Trotzek Kiesel (2019) Yin (2020) Proposed TBM
et al. et al. (2020) et al. (2018)
(2017b)

50 77.19 80.7 83.28 88.61 80.34 91.59

100 81.11 84.24 82.65 85.22 83.21 92.23
150 78.16 81.89 83.66 85.78 83.34 93.01
200 80.1 83.22 84.68 84.7 85.67 93.51
250 79.71 82.3 84.12 85.41 86.78 93.18

Table 4  Accuracy of proposed No.of users Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy
TBM model with different
works Vaswani Ramponi Trotzek et al. (2018) Kiesel (2019) Yin (2020) Proposed TBM
et al. et al.
(2017b) (2020)

50 82.28 85.38 88.66 94.22 89.75 94.23

100 876.26 88.57 89 90.5 90.66 95.89
150 82.27 86.19 88.06 90.3 88.98 96.12
200 84.14 87.17 89.75 89.43 90.12 96.75
250 84.52 87.68 89.49 88.98 89.84 97.03

Table 5  fMeasure of mood No.of users F1-Score F1-Score F1-Score F1-Score F1-Score F1-Score
classification for proposed
Vaswani Ramponi Trotzek et al. (2018) Kiesel (2019) Yin (2020) Proposed TBM
et al. et al.
(2017b) (2020)

50 76.97 80.47 83.05 88.36 87.43 92.33

100 80.89 84.01 82.42 84.98 82.67 93.69
150 77.94 81.66 83.43 85.54 83.98 92.91
200 79.88 82.98 84.44 84.46 85.72 93.25
250 78.94 82.31 84.16 84.94 86.77 92.99

strategies segment. The information was apportioned encouraging a comprehensive assessment. This technique
into three folds, each comprising of break even with rep- is vital for guaranteeing that comes about are not only
resentations of positive, negative, and unbiased tweets, artifacts of information distribution.
Int J Syst Assur Eng Manag

Table 6  Analysis of Analysis Proposed model TBM performance

performance
Train dataset Test dataset

Accuracy 0.94 0.96 0.97 0.78 0.97 0.99 0.99 0.88

Precision 0.73 0.8 0.81 0.87 0.82 0.87 0.86 0.97
Recall 0.85 0.86 0.89 0.89 0.87 0.89 0.93 0.95
F-Score 0.92 0.92 0.91 0.96 0.95 0.95 0.95 0.99

6.3 Performance metrics Based on these results it can be observed that the pro-
posed model is 7% more effective in terms of accuracy for
The execution of the proposed TBM is outlined in Tables 2, the same dataset.
3, 4 and 5, highlighting its prevalent execution in precision, The F1-Score, which harmonizes precision and recall, is
accuracy, review, and F1-score over shifting numbers of calculated as (Precision + Recall) / 2.
users. These parameters were evaluated and tabulated for ’N’
Precision is calculated as TP / (TP + 0046P), where TP number of users to gauge the system’s performance in
represents true positives, indicating the system’s correct detecting mental states.Based on these results it can be
detection of a positive mental state. TN stands for true nega- observed that the proposed model is 6% more effective in
tives, representing the system’s correct detection of a nega- terms of recall of mood classification for the same data-
tive mental state. FP is false positives, indicating instances set. Using these values fMeasure values were evaluated and
where the system incorrectly detects a positive mental state. tabulated in the following Table 5,
FN, on the other hand, represents false negatives, signifying The F1 score, which equalizations exactness and review,
the system’s incorrect detection of a negative mental state. encourage affirms the TBM’s strength, as shown in Table 5.
Based on these results it can be observed that the pro- An F1 score of 93.25% grandstands the model’s in general
posed model is 4% more effective in terms of accuracy for adequacy in disposition classification assignments, empha-
the same dataset. Table 2 shows that the proposed demon- sizing its utility in real-world applications.
strate accomplished a most extreme exactness of 92.89%, Table 6 summarizes the execution investigation, reflecting
outpacing the other models. This upgrade recommends that the TBM’s adequacy in both preparing and testing datasets.
the TBM successfully minimizes wrong positives, making it With precision levels coming to 0.99 and 0.97, separately,
a dependable choice for applications requiring tall precision. the show illustrates surprising consistency in its presci-
Due to this improvement in performance parameters, the ent capabilities over different scenarios. Furthermore, the
proposed model is well suited for real time implementation. accuracy and review values advance substantiate the model’s
Recall, which measures the system’s capacity to cor- strength, situating it as a reasonable arrangement for estima-
rectly identify positive mental states, is calculated as TP / tion examination errands.
(TP + FN). The proposed Transformer-based models have demon-
Using these measures, it can be observed that the pro- strated their effectiveness in sentiment analysis and can be
posed model showcases 4%–7% improvement in accuracy, customized and fine-tuned to suit specific applications and
precision, recall and fMeasure evaluations. Table 3 presents domains. Researchers and practitioners can leverage these
the review values, where the TBM come to 93.51% with 200 advanced models to achieve state-of-the-art results in senti-
clients, illustrating its capacity to distinguish a tall extent ment analysis tasks. The ability to predict users’ emotional
of genuine positive estimations. This capability is basic for behavior based on their social media data offers the poten-
applications where recognizing positive mental states is tial for advancing future wellness-enhancing technologies.
essential. Such technology could be instrumental in developing com-
Accuracy is assessed as (TP + TN) / (TP + TN + FP + FN), prehensive alert systems and mental health insights while
providing an overall measure of the system’s performance in safeguarding user privacy.
correctly detecting both positive and negative mental states.
As appeared in Table 4, the TBM reliably beated other mod-
els by 4% to 7%, accomplishing an noteworthy precision of 7 Conclusion and future scope
96.75% with 200 clients. This change is critical as it under-
scores the model’s capacity to generalize well over different In conclusion, the ability to predict users’ emotional behav-
datasets. ior based on their social media data offers the potential
for advancing future wellness-enhancing technologies.
Such technology could be instrumental in developing
 Int J Syst Assur Eng Manag

comprehensive alert systems and mental health insights Clark K, Luong MT, Le QV, Manning CD (2020) ELECTRA: Pre-
while safeguarding user privacy. For instance, authorities training text encoders as discriminators rather than generators.
Proceedings of the 8th international conference on learning
could choose to provide professional assistance or emotional representations (ICLR)
support, leaving the decision in the hands of consumers. Dai Z, Yang Z, Yang Y, Carbonell J, Le QV, Salakhutdinov R (2019)
In our view, it is imperative to address concerns regarding Transformer-XL: attentive language models beyond a fxed-
individual privacy and other ethical considerations when length context. arXiv preprint. arXiv:1901.02860
Dai X, Chalkidis I, Darkner S, Elliott D revisited transformer-based
analyzing social media content, as these issues are pertinent. models for long document classification.
This also raises questions about the utilization of potentially Devlin J, Chang MW, Lee K, Toutanova K, introduced BERT: bidi-
sensitive information, given its relevance to user behavior rectional encoder representations from transformers in their
and mental health. Transformer-based models have demon- 2018 arXiv preprint.
Devlin J, Chang MW, Lee K, et al presented BERT: pre-training of
strated their effectiveness in sentiment analysis and can be deep bidirectional transformers for language understanding in
customized and fine-tuned to suit specific applications and their 2018 arXiv preprint.
domains. Researchers and practitioners can leverage these Ding S and colleagues introduced ERNIE-Doc: a retrospective long-
advanced models to achieve state-of-the-art results in senti- document modeling transformer at ACL-IJCNLP 2021.
Dong H and colleagues explored explainable automated coding of
ment analysis tasks. clinical notes in JBI 2021.
This work can be extended by adding better facial features Gillioz J, Casas E, Mugellini, Khaled OA provided an overview of
for analysis, and via the use of fusion based deep learning transformer-based models for NLP tasks at the 2020 15th con-
models that can combine social data with video data in the ference on computer science and information systems (FedC-
SIS) in Sofia.
same architecture. Also, researchers can try addition of other Gururangan S and colleagues presented don’t stop pretraining: adapt
features like auditory responses, ecommerce patterns, food language models to domains and tasks at ACL 2020.
patterns, etc. and evaluate their effect on the performance. Gururangan S and colleagues shared insights on don’t stop pretrain-
ing: adapt language models to domains and tasks at ACL 2020.
Jia C, Yang Y, Xia Y, Chen YT, Parekh Z, Pham H, Le Q, Sung YH,
Li Z, Duerig T presented their work at the 38th international
Funding No funding received. conference on machine learning in 2021.
Jin B, Xu X (2024) Predictions of steel price indices through
Declarations machine learning for the regional northeast Chinese market.
Neural Comput Appl 36(33):20863–20882. https://​doi.​org/​10.​
Conflict of interest No Conflict of interest. 1007/​s00521-​024-​10270-7
Khanuja S, Tsvetkov Y, Salakhutdinov R, Black AW discussed
learning and evaluating emotion lexicons for 91 languages at
the 2019 annual meeting of the association for computational
linguistics.
References Kiesel J and others discussed hyperpartisan news detection in SemE-
val2019 task 4.
Aragón M, López-Monroy A, González-Gurrola L, Montes-Gómez M Li X, Xiong C, Callan J explored generating human-like responses for
delved into detecting depression in social media using fine-grained sentiment-based user simulations in the proceedings of the 57th
emotions at the 2019 conference of the North American chapter annual meeting of the association for computational linguistics
of the association for computational linguistics. in 2019.
Biswas S, Pal A, Mukherjee A (2020) Exploring deep learning models Liu S, Heinzel S, Haucke MN, Heinz A discussed increased psycho-
for twitter sentiment analysis. Proceedings of the 8th international logical distress, loneliness, and unemployment in the spread of
conference on advanced computing and Communication Systems COVID-19 over 6 months in Germany in their 2021 publication
(ICACCS) 1:146–151 in medicina.
Blitzer J, Dredze M, Pereira F (2006) Biographies, bollywood, boom- Rahali A, Akhloufi MA explored end-to-end transformer-based models
boxes and blenders: Domain adaptation for sentiment classifica- in textual-based NLP in AI 2023.
tion. Assoc Comput Linguist Ramírez-Cifuentes D, Freire A contributed to Upf’s Participation at
Brown T, and colleagues discussed language models are few-shot learn- the CLEF eRisk 2018: Early risk prediction on the internet, which
ers at the 34th annual conference on neural information process- was presented at the 9th international conference of the CLEF
ing systems in 2020.John Blitzer, Ryan McDonald, and Fernando association, CLEF 2018, Avignon, 2018.
Pereira. 2006. Domain adaptation with structural correspondence Ramponi A, Plank B conducted a survey on neural unsupervised
learning. In Proceedings of the 2006 Conference on Empirical domain adaptation in NLP in COLING 2020.
Methods in Natural Language Processing (EMNLP), pages 120– Sun C, Dehghani M shared insights on BERT for sentiment analysis
128, Sydney, Australia. Association for Computational Linguistics and dataset construction in their 2019 arXiv preprint.
Chalkidis I and colleagues presented an empirical study on large-scale Sun C, Qiu X, Xu Y, Huang X discussed how to fine-tune BERT for
multi-label text classification at EMNLP 2020. text classification in CCL 2019.
Chikersal P, Belgrave D, Doherty G, Enrique A, Palacios JE, Rich- Trotzek M, Koitka S, Friedrich C presented their work on word embed-
ards D, Thieme A presented their work titled understanding cli- dings and linguistic metadata at the CLEF 2018 tasks for early
ent support strategies to enhance clinical outcomes in an online detection of depression and anorexia at the 9th international con-
mental health intervention at the 2020 CHI CONFERENCE on ference of the CLEF association, CLEF 2018, Avignon.
human factors in computing systems.
Int J Syst Assur Eng Manag

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Yin W, Rajani NF, Radev D, Socher R, Xiong C shared their research
Polosukhin I, presented their groundbreaking work attention is on universal natural language processing with limited annotations
all you need in 2017. at the 2020 conference on empirical methods in natural language
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, processing (EMNLP).
Kaiser L, Polosukhin I presented their work titled attention is all
you need in 2017. Publisher’s Note Springer Nature remains neutral with regard to
Xia L, He L, Wan X (2021) Sentiment-based aspect-aware attention jurisdictional claims in published maps and institutional affiliations.
mechanism in deep learning for sentiment analysis. IEEE Trans
Neural Netw Learn Syst 32(3):1045–1059 Springer Nature or its licensor (e.g. a society or other partner) holds
Xiao L and colleagues presented label-specific document representa- exclusive rights to this article under a publishing agreement with the
tion for multi-label text classification in EMNLP-IJCNLP 2019. author(s) or other rightsholder(s); author self-archiving of the accepted
Xuetong C, Martin D, Thomas W, Suzanne E explored Identifying manuscript version of this article is solely governed by the terms of
depression on twitter with temporal measures of emotions at the such publishing agreement and applicable law.
web conference 2018.
Yang W, Luo L, Zheng K, Chen W, Xu K (2019) Sentiment analysis
and opinion mining of online reviews: A systematic literature
review. Expert Syst Appl 130:234–249