2024 2nd International Conference on Disruptive Technologies (ICDT)

Depression Detection Using Multimodal Analysis

with Chatbot Support
Archana Sharma1* Anuradha Saxena2 Ashok Kumar3
1,2,3,4Department of Computer Science 1,2,3,4Department of Computer Science 1,2,3,4Department of Computer Science
and Engineering and Engineering and Engineering
(ABES Institute of Technology, (ABES Institute of Technology, (ABES Institute of Technology,
Ghaziabad) Ghaziabad) Ghaziabad)
Affiliated to AKTU Lucknow Affiliated to AKTU Lucknow Affiliated to AKTU Lucknow
Up, India Up, India Up, India
[email protected] [email protected] [email protected]

Divyansh Singh
1,2,3,4
Department of Computer Science
and Engineering
2024 2nd International Conference on Disruptive Technologies (ICDT) | 979-8-3503-7105-5/24/$31.00 ©2024 IEEE | DOI: 10.1109/ICDT61202.2024.10489080

(ABES Institute of Technology,

Ghaziabad)
Affiliated to AKTU Lucknow
Up, India
[email protected]

Abstract— Depression, a widespread psychiatric disorder human beings becomes reduced in a depressed person [3].
affecting people globally, spans all age groups, predominantly These include verbal expressions, vocal intonations, and
impacting adults. This bipolar disorder characterized by written text, each serving as a distinct medium through which
symptoms including pessimism, hopelessness, anhedonia, and the emotional landscape of an individual can be deciphered.
sadness, significantly influences lives, contributing to As a result, the ability to recognize and interpret this diverse
depression. Our paper proposes a multi-model approach for array of cues, spanning both linguistic and non-linguistic
depression detection, utilizing facial expression analysis, audio dimensions of communication, holds the potential to offer
evaluation, and user text input through deep learning
profound insights into the mental state of individuals Various
algorithms, alongside an intelligent chatbot for personalized
traditional therapies and methods for detecting depression are
support. This hybrid model integrates facial expressions, audio
features, and textual input for a comprehensive approach to
time-consuming and expensive and most of them are
depression detection. The methodology includes four key ineffective. Some of these old therapies are such as
objectives: a CNN model for real-time or pre-recorded video psychotherapy or pharmacological [4]. One of the major
facial expression analysis, audio evaluation using an NLP problems with these types of traditional therapies is that they
algorithm to transcribe users' voices, text-based analysis require more information related to the patient's need, and
uncovering linguistic patterns and emotional context, and patient history for providing various types of information and
Multimodal Fusion integrating outputs for a unified multimodal require continuous monitoring of people's health activities.
approach. The intelligent chatbot encourages users to share And secondly, patients feel uncomfortable telling doctors
emotions openly, enhancing the system's accuracy in identifying about their mental condition due to their fear of society [5].
individuals at risk of depression. Results demonstrate the Various automatic systems are used for detecting depression
fusion's contribution to early depression detection, enabling in the early stages so that doctors can treat patients as soon as
timely interventions and improving accuracy, efficiency, and possible. Some of the tools needed earlier are an assessment
overall performance. system and an interview-style system. The assessment process
includes methods such as the Hamilton Rating scale and self-
Keywords— Multimodal Analysis, Pessimism, anhedonia, reporting techniques that contain the Beck Depression
influences, Linguistic patterns, Predominantly, Depression. Inventory tool and structured clinical interviews, and PHQ-8
I. INTRODUCTION scores that detect symptoms of depression and common action
in patients for detecting depression in the early stage of life
Depression is one of the most popular debilitating mental [6]. With the rise of various technologies Like Artificial
health problems that affects millions of people globally. WHO Intelligence, Machine Learning, Blockchain, and Fuzzy Logic
(World Health Organization) defines it as a persistent mental various techniques are formed that detect emotions in human
health condition that involves continuous sadness and beings very easily. Some papers detect depression using a
unhappiness in various activities.[1]. According to the survey Text-based system by using sentiment analysis for users'
of 2021 more than 280 million people are affected by various types of tweets and posts on various types of social
depression and for it, early detection of depression is networking sites [7]. Various machine learning algorithms
needed.[2] Depression, being a multifaceted disorder, often like Support Vector Machines SVM, KNN, LSTM, and
manifests through an intricate tapestry of signs and symptoms Naïve-Bayes. They are used for detecting depression and
encompassing diverse channels of communication. It mostly these algorithms need a confusion matrix for evaluating
affects an individual’s ability to learn anything and also causes results without using a confusion matrix they face problems in
various types of mood swings and it also reduces various analyzing results. In some research papers machine learning
capabilities of human beings as the working capacity of

979-8-3503-7105-5/24/$31.00 ©2024 IEEE 328

Authorized licensed use limited to: Amrita School of Engineering. Downloaded on October 09,2024 at 02:57:55 UTC from IEEE Xplore. Restrictions apply.
 2024 2nd International Conference on Disruptive Technologies (ICDT)

algorithms such as PCA (Principal Component Analysis) [8], facial expression[13] and the % accuracy level of this paper is
and KNN are used that helps to extract various facial features 70%.In this study, an automated modal is proposed that
for detecting depression these algorithms only extract facial collects data from patients' facial expressions and body
features LSTM, Linear Support Vector, RNN, and Logistic language and here machine learning algorithms are used that
Regression these algorithm helps to detect depression through extract various features that find the level of depression, and
text from various tweets [9]. Depression detection from accuracy is achieved here is 83.5%[14].
textual features or only from facial expressions does not
provide the correct level of depression. It does not provide B. Audio
high accuracy about the mental state of people and to remove According to the paper [15] an Artificial system is
this problem we provide an approach that provides a fusion of proposed that predicts depression by using various deep
CNN and NLP that provides a comprehensive understanding learning algorithms and user data is collected from audio,
of an individual’s mental state and also provides a chatbot that video, and speech, using these data predict mental disease, and
provides a personalized health support. Here In this paper, we here result is evaluated according to the confusion matrix. As
are using CNN and NLP for detecting depression which help per the study mentioned in [16] system is proposed that easily
in increasing accuracy for detecting depression and also helps detects depression through stress and mental conditions and
in the early detection of depression. then provides appropriate solutions and techniques to remove
that depression it's very helpful to users and is time-saving and
In this paper, we propose a novel multi-model approach manages depression very easily. The study mentioned in the
using CNN and NLP for detecting depression. We leverage [17] paper according to this research depression is detected
facial expression, audio processing, and user text input by using audio and speech using artificial intelligence in which
using deep learning algorithms. Additionally, an intelligent user’s behavior data is gathered through speech and audio and
chatbot is integrated to offer personalized support to users. the result is evaluated here using a confusion matrix. In the
The fusion of these modalities enhances the system's accuracy Research paper [18] depression is detected using speech
in identifying individuals at risk of depression. The empathetic signals and also depression from text-based languages and
chatbot's interaction contributes to early depression detection, uses Bi-LSTM for recognizing textual data and 1-D CNN for
timely interventions, and improved accuracy, efficiency, and extracting audio signals and it is one of the good ways but time
overall performance in depression detection. taken [19].
Here in this paper in Section 2, we discuss about literature C. Textual analysis( Twitter analysis and Social media
survey of work related to depression detection with analysis).
multimodal analysis using deep learning algorithms and in
Section 3, we show a proposed modal and algorithm for this According to the study [20], various levels of depression
proposed modal. In Section 4, we analyze the result and are by analyzing social media posts. In this research, two AI
experiment for this proposed modal. algorithms are used, SVM and Naïve-Bayes that classify user-
generated content from various social media platforms in this
II. LITERATURE SURVEY research paper dataset is collected from three social media
The literature survey of this research paper is partitioned platforms that are Facebook, Twitter, and Live Journal and
into three sections according to different domains (Face, Text, depression is detected in four stages these are ( Minimal, Mild,
Audio ). The first section described the detailed study of Moderate, High). Rapid Miner is used here for testing the
depression detection using facial expressions through trained classifiers and results are calculated by using a confusion
data or from pre-recorded videos. The second section matrix. Another accuracy is calculated by using recall and
discusses depression detection using audio processing. The precision accuracy is calculated that is 82%. The study
third or last section discusses depression detection using mentioned in the research paper[21] consists approach that
textual data. All sources of this different domain are described detects depression using an NLP algorithm from textual data
below in terms of using technology machine learning. and it uses lexicon terms that are nowadays very common to
very depressed people and result found in this study helps in
A. Facial Expression improving the accuracy and performance of the modal[22].
In the past decade, we have seen that there is a continuous From various types of social media posts, we can easily detect
increase in the number of depressed people and the main whether the patient is mentally depressed or not and also
domain in which depression can be detected is Facial easily find out they are facing which type of mental disorder.
expression. As per a study conducted in [10] automatic A solution proposed a modal in which depression is detected
recognition system was formed that recognizes facial features from social media platforms by using some artificial modal
for detecting depression study in this proposed system various like the MGL-CNN model [23]. This proposed modal predicts
various features such as Precision, Recall, accuracy, and F1-
processes take place pre-processing, feature extraction, and
score. Some research papers use ANN and DCNN for
other classifications and the system recognizes facial features
detecting depression in patients or to detect the mental state of
that are Happy, Angry, Fear, Neutral, Sad. The experimental
patients so that it helps in the early detection of depression in
study of various types of methods and techniques in [11] helps
patients [24]. The method used in [25] uses Natural Language
to extract and recognize features from facial expressions. in
Processing and sentiment analysis of users' tweets for
this found that face detection and extraction are performed on
detecting depression in this paper dataset is collected from
the proposed system. And it is one of the good ways to detect
social media posts by using APIs and then csv file is created
depression but only facial features do not produce accurate
for both training and testing dataset and also confusion matrix
depression in any person so the accuracy of this paper is
is formed for calculation of various results. As several
66%.In the study discussed in [12] here in this study
depressed people is increasing various models are proposed
depression is detected using facial expressions and video
for detecting depression but most of the solutions do not
using artificial intelligence and classification used here is the
achieve high accuracy or performance. Some researchers used
neural network and haar Cascade algorithm used for detecting

329
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on October 09,2024 at 02:57:55 UTC from IEEE Xplore. Restrictions apply.
 2024 2nd International Conference on Disruptive Technologies (ICDT)

social media platform results for predicting depression but C. Proposed Methodology
results produced from this platform are not accurate [26]. Only
social media or face do not predict accurate results. So the
combination of audio, video, and text features can accurately
predict depression accurate result.
III. PROPOSED WORK
The proposed model aims to develop a novel multi-modal
depression detection system using facial expression, audio,
and user text input and also provide an intelligent chatbot that
complements the detection system by giving individual
support to the user. This proposed Model is based on the two
most popular deep learning algorithms that are CNN
(Convolution Neural Network) and NLP (Natural Language
Processing). Deep learning algorithms are one of the best
solutions for detecting depression. Algorithms that are used Fig. 1. Explanations of System Architecture.
are described below:
“Fig.1” explains the proposed methodology that is used in
A. Convolutional Neural Network: the modal and it describes all the steps that are taken here to
CNN is a deep learning algorithm mainly used in image detect depression.
and video recognition and pattern recognition [27]. It contains
many features like it contain very simple structure and also Input: So the input is given in three ways facial expression,
having very less training parameters. It is based on the Audio, and text.
principle of Convolution operation and the process of this
Pre-Processing: Pre-processing is the second phase of the
convolution operation is explained here. Example- (Input
values* Kernal /filter(same size) = Output values (Feature system that includes quality enhancement in various types of
Map) One of the most important advantages of using CNN is signals that are mostly input signals of text. And also includes
that time required in processing is much lower in comparison signals that are audio speech here various types of unwanted
to the time that is required in other algorithms it working noise are removed and other unwanted things that are not
includes process that it takes Input in the form of image and required in these processes are removed.
assigns some weights to that input values and then work in Feature Extraction: Feature extraction is the phase that
recognition of the output values or we can define it as it is includes speech signal and also includes text that is given as
useful for finding patterns in images for identifying objects. input. Input that is given through speech includes various
Common Applications of CNN include pattern recognition, types of hidden information that mostly shows the emotional
text recognition, speech recognition, and text recognition, and condition. It is one of the most important processes that is
also used for understanding NLP problems. required for extracting the most important information related
to the required field and for the final result, it is needed that
B. Natural Language Processing(NLP). relevant features must be extracted from a given input.
NLP (Natural Processing Algorithm) algorithms play a
very crucial role in deep learning algorithms and detecting Classification: In the classification phase, the trained
depression by using textual data is known as “sentiment model is utilized to evaluate and categorize depression levels
analysis” or “emotion analysis”[28]. Here in this proposed based on multimodal input data [30]. The model utilizes
model, NLP processes user text input to uncover linguistic extracted features from facial expressions and user text input
patterns, sentiments, and emotional context related to their to make predictions regarding the severity of depression. The
mental state. And NLP component easily identifies and also result of this process can be expressed as either a probability
easily quantifies potential indicators of depression, in the field score, offering insights into the likelihood of depression, or
of mental illness. NLP plays a very important role in providing discrete classes that distinctly represent various levels of
various types of analysis and also providing management on a depression severity.
very large scale in the area of sentiment analysis, and D. Research Methodology.
information extraction [28]. There are various Applications of This research paper embarks on a multifaceted
NLP it is used for Text summarization, Used for Language exploration, venturing into the confluence of mental health,
Translation such as Google Translate, used for information technology, and human-computer interaction. Through the
extraction, and also used for text classification [29]. The integration of sophisticated NLP and CNN algorithms, we aim
resulting area of artificial intelligence has become very to penetrate the layers of linguistic subtleties and the subtle
popular for extracting various features and machines have emotional undercurrents embedded within user interactions
become very powerful for finding various functions and [31]. In addition to these technical aspects, the inclusion of
mechanisms that are used that extract depression very easily. chatbot support, designed to extend empathy and assistance to
Language plays a very crucial role in the life of human beings individuals in distress, stands as a pivotal element in our
and helps detect problems of various kinds. NLP is one of the approach. It is through this holistic fusion of cutting-edge
most important areas of artificial intelligence that is leading in technology and human compassion that we seek to propel the
the market for detecting various types of problems. frontiers of depression detection and mental health support.
This study proposes a modal that will be trained with various
types of features. These features include various trained
modals and conditions like happy, neutral, contempt, angry,

330
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on October 09,2024 at 02:57:55 UTC from IEEE Xplore. Restrictions apply.
 2024 2nd International Conference on Disruptive Technologies (ICDT)

sad, and fear [32]. Depression detection will take place in this F. Description of Trained Dataset.
proposed modal from the presence of these conditions in a Creation of a trained dataset for detecting depression in
human such as happiness, sadness, fear, neutrality, contempt, various ways like audio, video, and text is a very complicated
and some other features in capturing through video frames way and it requires various types of steps. At first, the dataset
after detecting depression it will be classified as having is gathered for the face by taking images of various facial
depression of different level like low, moderate and high. The conditions like happy, sad, neutral, fearful, and angry, and
architectural diagram describes the proposed automated various types of audio recordings are also gathered here for
modal that can be described or explained in the given diagram. creating the trained dataset. For video analysis, various video
E. Proposed Architectural Diagram: clips were gathered of various face conditions. A team of
various types of expert annotators and other reviewers detect
the presence and absence of depression from the Dataset very
easily [35]. Data that we gathered first go through the process
of preprocessing which includes the detection of depression
from the face, audio, or video, and also includes cropping
cutting, and editing of videos after that process of feature
extraction takes place that extracts features for identifying
depression and also video segmentation takes place.

Fig. 2. Architectural diagram for Proposed Modal.

Fig. 3. Training Dataset Creation
In “Fig. 2”, the Architectural diagram for the proposed
modal of depression detection using face audio and video is This proposed model extracts features from the video and
explained. analyzes the label of depression and a depressed person is a
In this project we analyze depression from images text and person who is very sad and not happy. “Fig. 3” explains about
videos, here in the given figure explains the system the trained dataset that we are using in this model. Facial
architecture of the project in which we gather the dataset expression is one of the best ways to detect depression because
multimodal, including images related to depressed or healthy it is non-verbal communication for showing the emotions of
people and text data related to symptoms of depression and any person or detecting the mental state of any person when
then in second step the pre-processing process takes place, in we make a comparison between FACS coding and Automated
the Pre-process the image data is used and we apply FACS coding for finding depression in a person’s facial
convolutional operations and pooling to extract relevant expression then it tells that there is the similarity between all
features from dataset And use rectified linear units (ReLU) as methods [36]. A person who is highly depressed has very little
activation functions to introduce non-linearity. The FCN smile on her face.
(fully connected network) Training is used to design and train G. Description of Testing Dataset.
a convolutional neural network [33]. And convolutional
Testing datasets for detecting depression in various ways
neural network is used for mainly Pre-processing the image
like from face, video, and audio required various types of steps
data. And FCN layer is also used here for capturing high-level
that evaluate the performance by using deep learning
features and fine the proposed modal to improve the efficiency
algorithms. In this proposed model various deep learning
and performance of the modal [34]. The trained model is used
algorithms are used for detecting depression from various
mostly for the trained CNN model for future predictions. Now
types of datasets. During the testing phase, mostly algorithms
for detecting depression from text analysis we first gather the
like deep learning algorithms take place on the training dataset
dataset. Then the text analysis process takes place and pre-
that is applied for that. Therefore, here in this way, depression
processing for text (NLP) takes place that firstly tokenizes and
is detected and performance is evaluated on the factors of
then cleans the text data and also applies various processes
various metrics such as precision, accuracy, and recall and
like natural language processing techniques, such as stemming
also includes F1 scores. So this evaluation helps in detecting
and lemmatization after pre-processing then classification
depression.
algorithm (Naïve Bayes) takes place that utilizes a naïve
Bayes classifier for analyze and checking the textual
symptoms now trained text model here used to save the trained
naïve Bayes model for future predictions. After that
integration of image and text analysis combines the output
from the image analysis branch and also from the text analysis
branch. Lastly fusion method is used in the project to merge
the results and create a unified representation. At last trained
model is used for predicting the final prediction of depression.

331
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on October 09,2024 at 02:57:55 UTC from IEEE Xplore. Restrictions apply.
 2024 2nd International Conference on Disruptive Technologies (ICDT)

Precision is expressed as the ratio of predictions that are

true positive to the total number of predictions that are positive
instance.
ȁȁ
ܲ‫ ݊݋݅ݏ݅ܿ݁ݎ‬ൌ
ȁܶܲȁ ൅ ȁ‫ܲܨ‬ȁ
Recall is defined that it is ratio of several predictions that
are true positive and classified to the number of actual positive
predictions. 
ȁȁ
ܴ݈݈݁ܿܽ ൌ
Fig. 4. Real-time camera detection ȁܶܲȁ ൅ ȁ‫ܰܨ‬ȁ
Here evaluation of the F1 score is performed by using the
given formula.
”‡ ‹•‹‘ ‫ŽŽƒ ‡ כ‬
‫ ͳܨ‬െ ‫ ݁ݎ݋ܿݏ‬ൌ ʹ ‫כ‬
ܲ‫ ݊݋݅ݏ݅ܿ݁ݎ‬൅ ܴ݈݈݁ܿܽ
By using the above formula following metrics are to
calculated.

TABLE I. COMPARISON TABLE OF DIFFERENT RESEARCH PAPERS.

Fig. 5. Images captured through camera

The above figures 4 and 5 display webcam images of a

person in real-time. The utilization of real-time camera data
for depression detection highlights its effectiveness as a
superior approach. This is evident through the application of
various deep-learning algorithms. We need a greater number
of training data samples for detecting depression, this helps to
predict depression easily and also increases the performance
and efficiency of depression detection [37]. Nowadays
depression has become an epidemic disease that affects all The table.1presents a comparison table of performance
stages of people and many algorithms and techniques are used metrics based on models used in the paper SVM used in the
for finding the level of depression, some of the algorithms paper [11], Naïve Bayes used in the paper [20] and LSTM
detect depression through Twitter, Facebook, Instagram, and used in the paper [27] and their corresponding indicators
also from eye moments of a person. Hence there are various determining the accuracy of the model and The table asserts
techniques and AI, and ML algorithms that detect depression that accuracy level of this paper is good and used multimodal
like RNN, CNN, NLP, PCA, KNN, SVM, and LSTM Naïve- analysis is one of the best approach for detecting depression
Bayes these algorithms show their crucial way in detecting the in comparison to others.
level of depression [38].
IV. RESULT ANALYSIS D^hZDEd
In Our comprehensive study of depression detection using WZEd'
multimodal analysis using face, video, and audio. The
proposed modal performance of all modalities is evaluated by DĞĂƐƵƌĞŵĞŶƚWĞƌĐĞŶƚĂŐĞ
using some parameters that are accuracy, precision, and recall ϵϰ͘ϮϮ
and also includes a score for this. In this proposed model CNN ϵϱ ϵϯ͘Ϯ
ϵϭ͘ϴϳ
achieved an accuracy of 92% along with its precision and ϵϬ͘ϰϭ
recall values. Similarly, by using NLP we found overall ϵϬ
accuracy of 92% from audio and video. The performance
parameter of the proposed system is taken here by using the ϴϱ
following indicators and these indicators help to evaluate the hZz WZ/^/KE Z>> &ϭ^KZ
accurate parameter of the proposed model.
Fig. 6. Measurement Percentage.
Accuracy is explained as that it contains a ratio of several
predictions that are correct to the total number of predictions. So it can be concluded that the fusion of CNN and NLP
It shows the truthfulness and correctness of the system. gives the best result for increasing the performance of the
system and give their best results. In ‘Fig.6”, a bar graph of
ȁȁ ൅ ȁȁ measurement of different performance metrics that shows the
‫ ݕܿܽݎݑܿܿܣ‬ൌ
ȁܶܲȁ ൅ ȁ‫ܲܨ‬ȁ ൅ ȁܶܰȁ ൅ ȁ‫ܰܨ‬ȁ accuracy level criteria is displayed. Overall, the developed
multi-modal depression detection system and chatbot hold
great potential to positively impact mental health support. The
fusion of facial expression analysis, NLP emotional analysis,

332
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on October 09,2024 at 02:57:55 UTC from IEEE Xplore. Restrictions apply.
 2024 2nd International Conference on Disruptive Technologies (ICDT)

and empathetic chatbot interactions can contribute to early providing personalized and empathetic support to individuals
detection, timely interventions, and enhanced mental well- in need. Through extensive data collection, model training,
being for individuals experiencing depression. and validation, the multi-modal system demonstrates
promising accuracy in identifying potential signs of
TABLE II. COMPARISON TABLE OF RMSE AND MAE depression, and also by leveraging real-time or pre-recorded
facial expressions and user text input, the system can offer
Modalities Methods Root Mean Mean timely interventions and personalized recommendations for
Square Absolute mental wellness resources. The chatbot's empathetic approach
Error (RMSE) Error (MAE) fosters a supportive environment, encouraging users to share
CNNs 8.49 7.62 their emotions and experiences openly. By providing relevant
Images SVM 9.49 8.42
Fuzzy Logic 9.20 8.62
resources like motivational videos, self-help books, and
RNNs 9.49 7.48 inspirational quotes, the chatbot empowers users to seek
CNNs 8.43 7.41 inspiration and motivation during challenging times. As
Audio MFCCs 7.93 8.20 further advancements and refinements are made, this
Fuzzy Logic 9.23 7.25 intelligent system can become an invaluable tool in the effort
GFCCs 7.22 8.92 to face the global challenge of detecting levels of depression
CNNs 7.40 8.20
Text RNNs 8.92 7.48
so that it shows the condition of mental condition. In the
Fuzzy Logic 9.02 8.44 future, there are several areas where improvements are needed
NLP 9.42 7.23 for the system. Firstly, there can be an increase in several
datasets that include different age groups and different stages
Audio + CNN + NLP 8.20 6.43 of depression. Secondly, an area of improvement can be in
Video + CNN +Fuzzy 8.49 6.90 the chatbot support system, in which we can provide online
Images Logic
CNN +SVM 9.23 7.49 counseling from various specialist doctors.
VI. REFERENCES
Now here in this table, RMSE, MSE, and MAE of
[1] Marriwala, N., & Chaudhary, D. (2023). A hybrid model for depression
different algorithms that are used in different papers for detection using deep learning. Measurement: Sensors, 25, 100587.
detecting depression are compared. These performance [2] Yadav, U., Sharma, A. K., & Patil, D. (2023). Review of automated
criteria like percentage error and root mean squared value depression detection: Social posts, audio and video, open challenges
(RMSE), also include others that are mean squared error and future direction. Concurrency and Computation: Practice and
(MSE) and also a percentage that are mean absolute Experience, 35(1), e7407.
percentages play their vital role in evaluating the accuracy of [3] Kavi Priya, S., & Pon Karthika, K. (2023). A contemporary multi-
the modal. As RMSE, MAE, and MSE all are inversely objective feature selection model for depression detection using a
hybrid pBGSK optimization algorithm. International Journal of
proportional to the accuracy it concludes that as much as these Applied Mathematics and Computer Science, 33(1).
values are lower accuracy is higher in the modal. For
[4] Kim, A. Y., Jang, E. H., Lee, S. H., Choi, K. Y., Park, J. G., & Shin, H.
evaluating these values following formulas are used here. C. (2023). Automatic 1. Squires, M., Tao, X., Elangovan, S.,
Gururajan, R., Zhou, X., Acharya, U. R., & Li, Y. (2023). Deep
Depression Detection Using Smartphone-Based Text-Dependent
Ψ‫ ݎݎܧ‬ൌ ȁ‫ݕ‬௜ െ ‫ݕ‬ഥȁȀ‫ݕ‬
ప ௜ ൈ ͳͲͲ Speech Signals: Deep Convolutional Neural Network
Approach. Journal of Medical Internet Research, 25, e34474.
‫ ܧܵܯ‬ൌ ͳȀܰ σே ௫
௜ୀଵሺ‫ ݕ‬െ ‫ݕ‬ሻ
ଶ
[5] Sharma, G., Joshi, A. M., Gupta, R., & Cenkeramaddi, L. R. (2023).
DepCap: A Smart Healthcare Framework for EEG-based Depression
Detection using Time-Frequency Response and Deep Neural
ே Network. IEEE Access.
ܴ‫ ܧܵܯ‬ൌ ඨͳȀ݊ ෍ ሺ‫ ݕ‬௫ െ ‫ݕ‬ሻଶ  [6] Rajawat, A. S., Bedi, P., Goyal, S. B., Bhaladhare, P., Aggarwal, A., &
௜ୀଵ
Singhal, R. S. (2023). Fusion Fuzzy Logic and Deep Learning for
Depression Detection Using Facial Expressions. Procedia Computer
The comparison of RMSE and MAE is described in Table, Science, 218, 2795-2805.
here in this table comparison takes the place of various [7] Chen, J., Hu, Y., Lai, Q., Wang, W., Chen, J., Liu, H., ... & Hu, X.
(2023). IIFDD: Intra and inter-modal fusion for depression detection
RMSEE and MAE values of different research papers so that with multi-modal information from the Internet of Medical
we can easily find out the difference between other research Things. Information Fusion, 102017.
that is taking place in this domain with various algorithms. [8] Guo, Y., Liu, J., Wang, L., Qin, W., Hao, S., & Hong, R. (2023). A
Overall, this multimodal approach provides a very easy and Prompt-Based Topic-Modeling Method for Depression Detection on
better approach for detecting depression in people with face, Low-Resource Data. IEEE Transactions on Computational Social
audio, and text methods and it is one of the best approaches Systems.
for identifying and addressing those persons who are suffering [9] Zhang, S., Zhang, X., Zhao, X., Fang, J., Niu, M., Zhao, Z., ... & Tian,
Q. (2023). MTDAN: A Lightweight Multi-Scale Temporal Difference
from mental health disorders. Attention Network for Automated Video Depression Detection. IEEE
Transactions on Affective Computing.
V. CONCLUSION
[10] Meshram, P., & Rambola, R. K. (2023). Diagnosis of depression level
The proposed research work aims to develop a novel using multimodal approaches using deep learning techniques with
multi-modal depression detection system using facial multiple selective features. Expert Systems, 40(4), e12933.
expression analysis and Natural Language Processing (NLP) [11] Thati, R. P., Dhadwal, A. S., Kumar, P., & P, S. (2023). A novel multi-
of user text input. By combining the strengths of CNN and modal depression detection approach based on mobile crowdsensing
and task-based mechanisms. Multimedia Tools and
NLP algorithms, the system can effectively capture emotional Applications, 82(4), 4787-4820.
cues and linguistic patterns indicative of depression. The [12] Fang, M., Peng, S., Liang, Y., Hung, C. C., & Liu, S. (2023). A
intelligent chatbot complements the detection system by multimodal fusion model with multi-level attention mechanism for

333
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on October 09,2024 at 02:57:55 UTC from IEEE Xplore. Restrictions apply.
 2024 2nd International Conference on Disruptive Technologies (ICDT)

depression detection. Biomedical Signal Processing and Control, 82, Users. ACM Transactions on Asian and Low-Resource Language
104561 Information Processing, 22(4), 1-19.
[13] Wu, P., Wang, R., Lin, H., Zhang, F., Tu, J., & Sun, M. (2023). [31] Xu, Y., Su, H., Ma, G., & Liu, X. (2023). A novel dual-modal emotion
Automatic depression recognition by intelligent speech signal recognition algorithm with fuses hybrid features of the audio signal and
processing: A systematic survey. CAAI Transactions on Intelligence speech context. Complex & Intelligent Systems, 9(1), 951-963.
Technology, 8(3), 701-711. [32] Shahzadi, I., Fuzail, M. M., & Aslam, N. (2023). Deep Emotions
[14] Iyortsuun, N. K., Kim, S. H., Jhon, M., Yang, H. J., & Pant, S. (2023, Recognition from Facial Expressions using Deep Learning.
January). A Review of Machine Learning and Deep Learning [33] Mamidisetti, S., & Reddy, A. M. (2023). A Stacking-based Ensemble
Approaches on Mental Health Diagnosis. In Healthcare (Vol. 11, No. Framework for Automatic Depression Detection using Audio
3, p. 285). MDPI Signals. International Journal of Advanced Computer Science and
[15] Hasib, K. M., Islam, M. R., Sakib, S., Akbar, M. A., Razzak, I., & Applications, 14(7).
Alam, M. S. (2023). Depression Detection From Social Networks Data [34] Hu, B., Tao, Y., & Yang, M. (2023). Detecting depression based on
Based on Machine Learning and Deep Learning Techniques: An facial cues elicited by emotional stimuli in video. Computers in
Interrogative Survey. IEEE Transactions on Computational Social Biology and Medicine, 107457.
Systems.
[35] He, L., Niu, M., Tiwari, P., Marttinen, P., Su, R., Jiang, J., ... & Dang,
[16] Yasin, S., Othmani, A., Raza, I., & Hussain, S. A. (2023). Machine W. (2022). Deep learning for depression recognition with audiovisual
learning based approaches for clinical and non-clinical depression cues: A review. Information Fusion, 80, 56-86.
recognition and depression relapse prediction using audiovisual and
EEG modalities: A comprehensive review. Computers in Biology and [36] Joshi, M. L., & Kanoongo, N. (2022). Depression detection using
Medicine, 106741. emotional artificial intelligence and machine learning: A closer
review. Materials Today: Proceedings, 58, 217-226.
[17] Muzammel, M., Salam, H., & Othmani, A. (2021). End-to-end
[37] Zogan, H., Razzak, I., Wang, X., Jameel, S., & Xu, G. (2022).
multimodal clinical depression recognition using deep neural
Explainable depression detection with multi-aspect features using a
networks: A comparative analysis. Computer Methods and Programs
hybrid deep learning model on social media. World Wide Web, 25(1),
in Biomedicine, 211, 106433.
281-304.
[18] Nash, C., Nair, R., & Naqvi, S. M. (2023). Machine Learning in ADHD
[38] Nadeem, A., Naveed, M., Islam Satti, M., Afzal, H., Ahmad, T., &
and Depression Mental Health Diagnosis: A Survey. IEEE Access.
Kim, K. I. (2022). Depression detection based on hybrid deep learning
[19] Ishimaru, M., Okada, Y., Uchiyama, R., Horiguchi, R., & Toyoshima, SSCL framework using self-attention mechanism: An application to
I. (2023). A New Regression Model for Depression Severity Prediction social networking data. Sensors, 22(24), 9775.
Based on Correlation among Audio Features Using a Graph
Convolutional Neural Network. Diagnostics, 13(4), 727. [39] Safayari, A., & Bolhasani, H. (2021). Depression diagnosis by deep
learning using EEG signals: A systematic review. Medicine in Novel
[20] Guiñazú, M. F., González, M., Ruiz, R. B., Hernández, V., Diez, S. B., Technology and Devices, 12, 100102.
& Velásquez, J. D. (2023). A novel depression risk prediction model
based on data fusion from Chilean National Health Surveys to diagnose
risk depression among patients with mood disorders. Information
Fusion, 100, 101960.
[21] Jadhav, G., Babar, S., & Mahalle, P. (2023, March). A Survey:
Performance-aware Depression Detection. In 2023 10th International
Conference on Computing for Sustainable Global Development
(INDIACom) (pp. 1242-1249). IEEE.
[22] Yang, L., Zhang, J., Yu, J., Yu, Z., Hao, X., Gao, F., & Zhou, C. (2023).
Predicting plasma concentration of quetiapine in patients with
depression using machine learning techniques based on real-world
evidence. Expert Review of Clinical Pharmacology, 16(8), 741-750.
[23] Sardari, S., Nakisa, B., Rastgoo, M. N., & Eklund, P. (2022). Audio-
based depression detection using Convolutional Autoencoder. Expert
Systems with Applications, 189, 116076.
[24] Oh, J., Kim, M., Park, H., & Oh, H. (2023). Are You Depressed?
Analyze User Utterances to Detect Depressive Emotions Using
DistilBERT. Applied Sciences, 13(10), 6223.
[25] . Nash, C., Nair, R., & Naqvi, S. M. (2023). Machine Learning in
ADHD and Depression Mental Health Diagnosis: A Survey. IEEE
Access.
[26] Milintsevich, K., Sirts, K., & Dias, G. (2023). Towards automatic text-
based estimation of depression through symptom prediction. Brain
Informatics, 10(1), 1-14.
[27] Xu, X., Wang, Y., Wei, X., Wang, F., & Zhang, X. (2023). Attention-
Based Acoustic Feature Fusion Network for Depression
Detection. arXiv preprint arXiv:2308.12478.
[28] de Lope, J., & Graña, M. (2023). An ongoing review of speech emotion
recognition. Neurocomputing, 528, 1-11.
[29] Cohen, J., Richter, V., Neumann, M., Black, D., Haq, A., Wright-
Berryman, J., & Ramanarayanan, V. (2023). A multimodal dialog
approach to mental state characterization in clinically depressed,
anxious, and suicidal populations. Frontiers in Psychology, 14.
[30] Duwairi, R., & Halloush, Z. (2023). A Multi-View Learning Approach
for Detecting Personality Disorders Among Arab Social Media

334
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on October 09,2024 at 02:57:55 UTC from IEEE Xplore. Restrictions apply.