2023 International Conference on Computer Communication and Informatics (ICCCI ), Jan.

23 – 25, 2023, Coimbatore, INDIA

Music Recommendation Based on Facial Expressions

and Mood Detection using CNN
Ganne Vaishnavi R.Sumathi, Karnatakam Anvitha,
Department of Computer Science and Assistant Professor, Department of Computer Science and
2023 International Conference on Computer Communication and Informatics (ICCCI) | 979-8-3503-4821-7/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICCCI56745.2023.10128353

Engineering Department of Computer Science and Engineering

Kalasalingam Academy of Research Engineering Kalasalingam Academy of Research
and Education Kalasalingam Academy of Research and Education
[email protected] and Education [email protected]
[email protected]

Dhathri Bathineed . Balabhadra Nikhitha .Kandregula Vanaja

Department of Computer Science and Department of Computer Science and Department of Computer Science and
Engineering Engineering Engineering
Kalasalingam Academy of Research Kalasalingam Academy of Research Kalasalingam Academy of Research
and Education and Education and Education
[email protected] [email protected] [email protected]

Abstract— Music is considered as global language. That can

connect the people in all over the world. Music has the great II. RELATED WORKS
ability to evoke different emotions in people. Music has great
impact on our emotions. It is also a means of mood regulation. In
the proposed system, music playlist is created by detecting Anima majumder et.al[1] proposed an automatic facial
emotions. A user’s emotion is detected by people facial expression approach based on deep network frame work with
expressions. The facial expressions can be detected lively by four inputs which includes geometric features extraction,
system’s camera. The convolutional Neural Network is used for Regional local binary pattern, fusion using auto encoders and
emotion detection. Pygame and Tinker are used for music classification using kohonen self-organizing map. Based on
recommendation. Feature extraction is performed on input
images to detect emotions such as happy, sad, chill. A music
geometric features with auto encoders the facial expression in
playlist is created based on our mood various types are detected with 98% classification accuracy
and the integration of K-SOP with regional local binary
Keywords- Universal, regulation, convolutional neural network, pattern is better than various existing techniques. Abishek
pygame, tinker Paudel and Miran Ghimire [2] used the creation of music
playlist based on user’s personality. It is to determine the
I. INTRODUCTION personality of a person on social media like instagram, twitter,
Facial Expression is the way of expressing emotions and there facebook can be one of platform where the people express
are different kinds of emotions like chill, sad, happy etc .This their feelings and opinions on different issue. Expressions of
type of emotions of a human kind can be detected by facial thoughts and opinions can be used to study the personality
expression. The best way in which people tend to analyse the traits. personality traits of the person can be studied in terms
emotion or the feeling or thoughts is trying to express by facial of standard big five responsibility traits define as openness to
expressions Music is the best medicine to alter the mood of an experience, conscientiousness, extraversion, agreeableness and
individual. Music has the ability to lighten our mood, calm neuroticism. python, django, numpy, pandas, nltk, html/css,
mind Music has a great impact on one’s mood. Our proposed Javascript, post gre sql, git are used for technologies and tools.
system with music recommendation based on facial expression logistic regression model and naïve bayes’s is used for
and mood detection based on various leading approaches like creating confusion matrix of big five personality classes.
deep learning, machine learning and neural networks. It can Aengcheng Wei et.al [3] utilized, the previous authentication
capture the facial expression and recognize the emotion of a methods like pattern, pin or password upgradation to the
person and create a playlist matching the one’s mood. The fingerprint, facial recognition etc. in case of using the facial
facial expression can be captured through web cam available recognition, one should be able to understand the features of
on computers android apps. For music recommendation we the face as there may be similar kind of faces but with
use libraries like pandas, openCV etc. different features. features must be observed to differentiate
the persons, many methods can be used for analysis of
features. Principal component analysis, Linear discriminant
analysis, independent component analysis can be used to group
the similar features, so that different features cannot be under

979-8-3503-4821-7/23/$31.00 ©2023 IEEE

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:23:57 UTC from IEEE Xplore. Restrictions apply.
 2023 International Conference on Computer Communication and Informatics (ICCCI ), Jan. 23 – 25, 2023, Coimbatore, INDIA

one group, the similarity of features in all the groups can be assigned for that particular emotion is played and in the same
avoided. Eventually, different features can be extracted easily way the songs are played for other emotions .It is just by
by grouping the similar features of similar so that the features detecting the facial expression and emotion of the user. This
can be differentiated. Adiyansjah, Alexander a s Gunawan, will give the result in the recommended playlist for the user.
Derwin Suhartono [4] proposed convolutional recurrent neural We used a library called as pygame for playing the audio.
networks for the recommendation of music. the task is that Function of the library is used to work with the music player.
automatic playlist has to be created by recognizing the For the developing GUI we can use Tkinter. Figure 1 briefs
emotion of human at particular situation. Music the flow work of our proposed work,
recommendation system can be divided into three parts users,
items and user-item matching algorithm. Buetal – used social C. Conventional Neural Network
media information to give more accurate music
recommendations. Bogdanov et al [5] exploited genre CNN is one of the famous NN algorithms which detect the
metadata to increase listener satisfaction based on face emotion in efficient manner [6]. It aids to predict the
collaborative filtering and content-based filtering. moods based the features, even though it is not a good
collaborative filtering method-sparse assessment matrix as the learning features but good in extract the high level features
users may listen only some part or some libraries, as the effect like edges, colour and textures.
of this, most of the assessments are not determined. Contact-
based filtering is a two-stage approach.
Image
III . METHODOLOGY START Start Recognition
camera
The music recommendation is based on the facial expression
and that implements the mood detection. It mood detection
consists of two parts: 1. Facial expression Detection 2.Music No
Recommendation Detection. Face
Detection

A.FACIAL EXPRESSION DETECTION:

Yes
The python libraries are used in it. The python libraries Filter
detects the various number of faces with bitmap graphic music
object and provides the count of all faces present in any based on Music Emotion
images. Various emotions of person like happy, surprise and language Recommendation Detection
fear by using facial expression detection. To classify the
emotions for this task and keros module of python is used for
it. But it takes a lot of time to work with android apps. The
Conventional neural network is aids for classification of
various types of emotions in any input image. We validate our
Display playlist
model by using publicly available datasets like Real-world
Affective Faces Database with training 80% and testing
20%.These dataset has some set of images and it contains
different specifications in it and all these images are converted Play Music EXIT
and combined to obtain a larger dataset with training datasets.

B.MUSIC RECOMMENDATION MODULE :

As per the person mood the Dataset is classified. In these Data Figure 1 Flow Diagram of System
set contain the songs according to the mood. These datasets
was found on Cohn-Kanade (CK) database with various
emotions. Data has been stored in cloud storage module. IV. RESULT AND DISCUSSION
These Data has arranged as per users request and In this Overall universe many peoples having similar faces and it is
process the ANS, google cloud were found but they are very hard to detect the human emotion efficiently. By using
rejected because, they are costly and limited storage. After facial expression we can detect the emotion up to certain level.
doing a lot of research the Firebase is considered as It can be used in the Android apps and also in the computer
blockserver. It can be in android app also. The music is played systems by using some applications. The camera can capture
according to mood detect by the picture. And the music is the image and recognize the face in the image and then detect
played by music recommended to that mood in the playist. the emotion in the image. Various faces with emotions are
When the surprise emotion is detected the songs that are shown in figure 2.

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:23:57 UTC from IEEE Xplore. Restrictions apply.
 2023 International Conference on Computer Communication and Informatics (ICCCI ), Jan. 23 – 25, 2023, Coimbatore, INDIA

Performance metrics:

Accuracy is used to measure the performance of the model

Figure 2. Sample Face emotion recognition images

With available dataset and collected from online dataset like

kaggle (400), we detect the emotions of faces into happy, sad,
angry and fear and based on the emotions music playlist is
recommended .We split 70 % for training anf 30% for testing
for validating the outcome[7-8]. The CNN parameters used for
our detection is mentioned in the following table 1.

Table 1: CNN parameters

Parameters name Size

Kernel size 3x3
Activation ETU
Kernels 128
Padding Same

Figure 4 : Comparison with existing techniques

From the Figure 4 it was proved that our CNN method is more
suitable for detecting the mood of any person with 92%
validation accuracy whereas SVM yields 65% and ELM yields
64%. With the open CV implementation we proved that
detecting the emotion and categories the music play list based
on the mood of captured images with CNN in efficient
manner.

V. CONCLUSION
Human emotions are very difficult to categorize and to detect
the mood based on the facial expressions and emotions it is
Figure 3: real facial expression and their recommended music definitely a challenge to us but with the help of the neural
playlist network and machine learning this model can be trained and to
get the maximum accuracy. Expressions are different from
person to person so, it is difficult to detect a emotion of a
With CNN model it detects the facial expression of various person. we have to train the model with larger number of
faces is shown in Figure 3. Based on the expression and mood images and we get more accurate with image datasets. Mostly
recommended music list is displayed, based on the list we can the major moods such as anger, disgust, fear, happy, sad,
able to categorizes the playlist for finding the trends of music surprise and neutral can be detected to 92% accurately. Face
for all age peoples. recognition is the primary step of the project. Without the face

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:23:57 UTC from IEEE Xplore. Restrictions apply.
 2023 International Conference on Computer Communication and Informatics (ICCCI ), Jan. 23 – 25, 2023, Coimbatore, INDIA

expression this project is nothing so this is the first step in it

and later this image processing can using the neural networks
to the given dataset, finally mood is detected which further
recommends the songs based on it. Now a days, Rather than
depending only on the facial expressions some other
additional factors included in it like heart rate. It could be
considered to detect moods like fear and surprise by heart rate.
The music recommendation is depend on the taste of the
person It can also be included in this model like
recommending the songs which are more often chosen by the
person according to the mood category. So this project has a
great future scope as it could be further extended to
recommend series, shows, movies and many more.

VI.REFERENCES
[1]Anima Majumder, Laxmidhar Behera, Venkatesh K.
Subramanian,"Emotion recognition from geometric facial features using self-
organizing map", Pattern Recognition, Vol 47, No.3, pp 1282-1293, 2014

[2] A.Paudel, B. R. Bajracharya, M. Ghimire, N. Bhattarai and D. S. Baral,

"Using Personality Traits Information from Social Media for Music
Recommendation," 2018 IEEE 3rd International Conference on Computing,
Communication and Security (ICCCS), pp.116-121, 2018

[3]Pengcheng Wei, Fangcheng He, Jing Li:

“Fast detection of moving objects based on sequential images
processing”, Journal of Intelligent fuzzy system. Vol. 39, No.4, pp 5037-5044,
2020

[4] Adiyansjah, Alexander A S Gunawan and Derwin Suhartono. “Music

Recommender System Based on Genre Using Convolutional Recurrent Neural
networks”, Procedia Computer Science, Vol.157, pp 99-109, 2019

[5] Dmitry Bogdanov, Martín Haro, Ferdinand Fuhrmann, Anna Xambó,

Emilia Gómez, Perfecto Herrera, “Semantic audio content-based music
recommendation and visualization based on user preference examples”,
Information Processing & Management, Vol 49, No.1, pp 13-33,2013

[6] Rabia Qayyum, Vishwesh Akre, Talha Hafeez, Sheeraz Ahmed, Asif
Nawaz, Hasan Ali Khattak, Pankaj Mohindru, Doulat Khan, Khalil Ur
Rahman, “Android based Emotion Detection Using Convolutional Neural
Networks”, 2021 International Conference on Computational Intelligence and
Knowledge Economy (ICCIKE) ,2021

[7] Ziyang Yu , Mengda Zhao , Yilin Wu , Peizhuo Liu , Hexu Chen,

“Research on Automatic Music Recommendation Algorithm Based on Facial
Micro-expression Recognition”. Proceedings of the 39th Chinese Control
Conference July 27-29, 2020, Shenyang, China.

[8] Miquel Mascaró Oliver, Esperança Amengual Alcover, UIBVFED: Virtual

facial expression , April 6, 2020.

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:23:57 UTC from IEEE Xplore. Restrictions apply.