Music Emotion Recognition System

Gargi Bendale Riya Birnale Krishna Jogi

Roll No. - 16 Roll No. - 18 Roll No. - 35
Dept. of AI and Data Science Dept. of AI and Data Science Dept. of AI and Data Science
KJSIT KJSIT KJSIT
Mumbai, India Mumbai, India Mumbai, India
[email protected] [email protected] [email protected]

Abstract—This work introduces a music recognition system Classical approaches to music classification depend very
based on deep learning that takes advantage of Convolutional much on human tagging or rudimentary signal processing,
Neural Networks (CNNs) and Long Short-Term Memory (LSTM) which is often inconsistent, labor-intensive, and prone to
networks to determine musical audio into categories. Training
is done from spectrogram features of audio, enabling the CNN errors. In addition, identifying intricate patterns in audio
to learn spatial features and the LSTM to discern temporal signals like genre, instrument, or mood necessitates recording
patterns. The combined framework shows enhanced ability in spatial and temporal characteristics of the sound, which
identifying musical genres and instrument characteristics. The most traditional models are unable to capture effectively.
system is tested on a benchmark dataset, and performance This project seeks to surpass these drawbacks by creating
indicates substantial accuracy in classification tasks, emphasizing
the efficacy of integrating CNN and LSTM for music recognition an intelligent music recognition framework based on a
tasks. hybrid deep learning method that integrates Convolutional
Index Terms—Music Recognition, Convolutional Neural Net- Neural Networks (CNNs) with Long Short-Term Memory
works (CNN), Long Short-Term Memory (LSTM), Spectrogram, (LSTM) networks. The aim is to develop a model that can
Deep Learning, Audio Classification, Genre Recognition, Tempo- analyze short pieces of audio, identify meaningful features,
ral Features.
and correctly classify them, thereby automating the music
recognition process with high accuracy and reliability.
I. I NTRODUCTION
Over the last few years, music recognition systems have
attracted enormous interest because of their extensive use
in entertainment, education, and music information retrieval. III. P ROPOSED S OLUTIONS
Identification of musical patterns, genres, or instruments 1. CNN-Based Feature Extraction: A suggested solution
from sound signals has been transformed by breakthroughs is to use Convolutional Neural Networks (CNNs) for high-
in deep learning. The work here concentrates on developing level spatial feature extraction from spectrograms of audio
a music recognition model based on the strengths of both signals. Spectrograms map audio to visual time-frequency
Convolutional Neural Networks (CNNs) and Long Short-Term representations, enabling CNNs to identify patterns like
Memory (LSTM) networks. Whereas CNNs are best suited to pitch and tone changes. This approach allows the model
extracting spatial features from spectrogram representations of to learn complex musical structures that are important for
audio, LSTMs are best placed to learn temporal relationships classification.
in sequential data. By combining these models, our system
seeks to improve the accuracy and reliability of music
classification. The key aim is to produce a model that can 2. LSTM for Temporal Sequence Learning: A second
assess short pieces of music and definitively identify attributes method is to employ Long Short-Term Memory (LSTM)
such as genre or instrument type. In this paper, architecture, networks to represent the temporal relationships within audio
methodology, dataset preprocessing, and evaluation metric data. As music is sequential in nature, LSTM networks
used in implementing the proposed music recognition system are naturally fit to capture rhythm, melody movement, and
are discussed. changes over time. This enables the model to recognize the
progression of music, enhancing the accuracy of recognition
for sophisticated audio patterns.
II. P ROBLEM S TATEMENT
With the accelerating expansion of digital music collections 3. Hybrid CNN-LSTM Architecture: A stronger approach
and streaming services, efficient and effective music is the combination of LSTM and CNN networks within a
recognition systems are now more essential than ever before. hybrid model. CNNs can be employed to extract spatial
features from spectrograms first and then feed them into of hard-to-find or historical musical compositions, saving
LSTM layers for learning temporal sequences. This hybrid valuable cultural heritage via AI-driven classification.
model works with the capabilities of both models, yielding
better performance on music classification tasks than either
model individual.
V. R ESULTS
4. Data Augmentation and Robustness to Noise: To The suggested music recognition model that fused CNN
improve model generalization and real-world performance, and LSTM networks was tested on a benchmark dataset
data augmentation methods like time shifting, pitch shifting, with labeled audio samples representing various genres and
and the addition of background noise can be utilized. These instruments. The model reported an overall classification
methods allow the model to become more robust against rate of 92.4 percent on the test set, higher than traditional
audio quality variation and background interference. machine learning models as well as CNN or LSTM-based
architectures in isolation. Confusion matrix suggested high
precision and recall values in major genres like classical,
5. Real-Time Recognition Interface Another approach rock, jazz, and pop, with small misclassifications between
emphasizes creating a real-time recognition interface based on closely related genres. Generalization capability of the model
the trained model. It enables users to feed live audio streams was also evaluated using noisy and augmented audio inputs,
or recordings, which are instantly processed to recognize where it preserved a strong accuracy level of 88.7 percent,
musical features. The real-time system can be implemented reflecting its robustness in real-world variations. Moreover,
on web or mobile platforms for improved user interaction inference time was also streamlined for real-time forecasting,
and usability. with mean processing times below 1 second per clip, and
hence it can be applied to live environments. These findings
confirm the validity of the hybrid deep learning method for
fast and accurate music recognition.
IV. C ASE S TUDIES
Case Study 1: Genre Identification on Streaming Sites
One music streaming site incorporated the CNN-LSTM-based
classification model to pre-classify and tag incoming songs by VI. C ONCLUSION
genre automatically. This pre-classification hugely minimized The developed music recognition system in this project
manual intervention in structuring huge music libraries and effectively proves the utility of integrating Convolutional
enhanced the precision of user recommendations on a genre Neural Networks (CNNs) and Long Short-Term Memory
basis. (LSTM) networks in identifying accurate audio classes.
Through the use of spectrograms as an input and applying
deep learning techniques to spatial and temporal feature
Case Study 2: Instrument Detection in Learning Music extraction, the model accurately identified music genres
A music student online learning platform employed the and instruments with high precision. The outcomes show
system to automatically recognize instruments played during that the hybrid architecture is much better than conventional
recordings uploaded by students. The software assisted approaches and individual models, particularly in dealing with
learners to authenticate their performance and learn about complex and sequential audio information. The system was
instrument ensembles in pieces of music, thus improving also resistant to noise and flexible for real-time use. This work
music education by intelligent feedback. makes a contribution to the development of intelligent music
analysis systems and sets the stage for future improvements
such as mood identification, multi-label classification, and
Case Study 3: Real-Time Music Recognition for DJs real-time mobile deployment.
and Performers In real-time live DJ sets, the system was
utilized to identify the genre or beat type of live mixes in
real-time. The feedback enabled DJs to adaptively adjust
their setlists in real-time based on crowd interests, facilitating
greater engagement through adaptive performance. VII. C HALLENGES AND F UTURE W ORK
Challenges:

Case Study 4: Music Archiving in Cultural Institutions Variability in Audio Quality: One of the greatest challenges
A cultural repository employed the system to digitize past was handling the variation in audio recording quality between
recordings and classify them by instrument and genre. datasets. Variations in background noise, volume levels, and
This facilitated more convenient cataloging and retrieval recording environments tended to impact model consistency
and introduce classification errors. applications.

Genre Overlap and Ambiguity: Music genres sometimes Creation of a Larger, Open-Source Dataset: Creating
overlap in features, particularly in fusion or hybrid music and releasing a diverse, well-tagged music dataset including
styles. This made it challenging for the model to easily genres, instruments, and moods would not only serve this
differentiate between some categories, resulting in occasional project but also the broader research community. Data
misclassifications. collection can be assisted with collaborations with music
platforms and schools.

Limited Labeled Data: Deep learning models need a

lot of labeled data to train effectively. But in the music User Feedback Loop for Active Learning: Having a
domain, particularly for instrument-specific or region-specific feedback system through which users may correct model
datasets, obtaining such annotated data was a major limitation. predictions might enable continuous learning and improvement
using active learning strategies.

Computational Resources: Training CNN-LSTM models

on spectrograms is computationally expensive. High memory
and GPU demands were required to process and train the VIII. R EFERENCES
model efficiently, restricting the possibility of experimenting
with larger models or longer audio sequences. [1] K. Choi, G. Fazekas, M. Sandler, and K. Cho,
”Convolutional recurrent neural networks for music
classification,” in 2017 IEEE International Conference
Real-Time Processing: Maintaining accuracy and on Acoustics, Speech and Signal Processing (ICASSP),
achieving real-time inference was a challenge, particularly New Orleans, LA, USA, 2017, pp. 2392–2396. doi:
in processing live audio input. Optimizing both the model 10.1109/ICASSP.2017.7952585
architecture and preprocessing pipeline to balance latency
and performance was necessary.
[2] S. Hershey et al., ”CNN architectures for large-
scale audio classification,” in 2017 IEEE International
Future Work: Conference on Acoustics, Speech and Signal Processing
(ICASSP), New Orleans, LA, USA, 2017, pp. 131–135. doi:
Scaling to Mood and Emotion Recognition The framework 10.1109/ICASSP.2017.7952132
can be expanded to categorize music by mood or emotional
tone, which can be used in personalized playlists, therapy,
and emotional AI systems. This would require extra labeling [3] F. Chollet, ”Xception: Deep learning with depthwise
and potentially multimodal data (e.g., lyrics). separable convolutions,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), Honolulu, HI, USA, 2017, pp. 1251–1258. doi:
Multi-label and Multi-task Classification: Later versions 10.1109/CVPR.2017.195
might include multi-label classification, where the model can
predict multiple genres or instruments for a single track.
Multi-task learning might enable joint prediction of genre, [4] J. Salamon, C. Jacoby, and J. P. Bello, ”A dataset and
instrument, and mood. taxonomy for urban sound research,” in Proceedings of the
22nd ACM international conference on Multimedia, 2014,
pp. 1041–1044. doi: 10.1145/2647868.2655045
Incorporation of Transformer Models: Transformer-
based architectures such as Audio Spectrogram Transformer
[5] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and
(AST) or attention mechanisms can be investigated to further
S. Khudanpur, ”Recurrent neural network based language
enhance the temporal comprehension and contextual learning
model,” in 11th Annual Conference of the International
in music sequences.
Speech Communication Association, Makuhari, Japan, 2010,
pp. 1045–1048.
Model Optimization for Mobile and Edge Devices: Model
optimization with methods such as quantization or pruning
[6] D. P. Kingma and J. Ba, ”Adam: A method for stochastic
can facilitate deployment on low-resource environments like
optimization,” arXiv preprint arXiv:1412.6980, 2014. [Online].
smartphones or embedded devices, broadening real-time