Java SB
Java SB
A PROJECT REPORT
Submitted by
of
BACHELOR OF ENGINEERING
in
APRIL 2020
BONAFIDE CERTIFICATE
Certified that this Report titled “COUNTENANCE BASED HARMONY” is the bonafide
work of “SIVA KARTHICK M (210416104101), NARESH ANAND M
(210416104065)” who carried out the work under my supervision. Certified further that to
the best of my knowledge the work reported herein does not form part of any other thesis
or dissertation on the basis of which a degree or award was conferred on an earlier occasion
on this or any other candidate.
(SIGNATURE) (SIGNATURE)
Dr.B.Sundarambal., Dr.R.Janarthanan, M.Tech,MBA,P.hD.
HEAD OF THE DEPARTMENT SUPERVISOR
Professor Professor
Dept. of Computer Science and Dept. of Computer Science and
Engineering Engineering
Chennai Institute of Technology Chennai Institute of Technology
Kundrathur-600069 Kundrathur-600069
i
ACKNOWLEDGEMENT
We convey our profound thanks and gratitude to our honorable Chairman
Mr.P.Sriram, Chennai Institute of Technology, Chennai for providing an
excellent academic climate, which made this endeavor possible.
We also express our gratitude to our beloved Principal Dr.P.Parthiban., who
constantly nurtured our standard of education and devoted their precious time for
our needs.
We deeply indebted to pay our sincere thanks to our respectable Head of the
Department Dr.B.Sundarambal., Department of Computer Science and
Engineering and Supervisor Dr.R.Janarthanan.,M.Tech,MBA,Ph.D., Department
of Computer Science and Engineering for showing some exuberant consent for our
project and providing us with all the facilities in the department to complete the
project.
We thank all our department teaching and non-teaching staffs, who have
helped directly and indirectly to complete this project in time.
Last but not least we extend my deep gratitude to our beloved family members
for their moral coordination, encouragement and financial support to carry out this
project.
Finally we express our heartfelt and deep sense of gratitude to all faculty
members in our division and to our friends for their helping hands, valuable, support
and encouragement during the project work.
ii
ABSTRACT
The user’s emotion using its facial expressions will be detected. These
expressions can be derived from the live feed via system’s camera or any pre-
existing image available in the memory. Emotions possessed by humans can be
recognized and has a vast scope of study in the computer vision industry upon which
several researches have already been done. The work has been implemented using
Python (3.6), Open Source Computer Vision Library (OpenCV) and NumPy. The
scanned image (testing dataset) is being compared to training dataset and thus
emotion is predicted. The objective of the system is to analyze the image and predict
the expression of the person.
The FisherFaces algorithm can be used to detect the face from the image
through average classification. The emotion can be detected by taking various
images of different emotions to train the data-set. The model can be trained for
improved accuracy and robust. Once the emotion is successfully detected with some
basic error reductions, the music can be played or suggested.
iii
TABLE OF CONTENTS
CHAPTER TITLE PAGE NO
ABSTRACT iv
LIST OF FIGURES vii
LIST OF ABBREVIATIONS viii
1 INTRODUCTION 1
1.1 Objective 1
1.2 Definition of Image 2
1.3 Image Processing 3
1.4 Types of Image Processing 4
1.5 Application of Image Processing 8
1.6 Face Detection 14
1.7 Emotion 18
2 SYSTEM ANALYSIS 23
2.1 Existing System 23
2.2 Literature Survey 27
2.2 Proposed System 35
3 REQUIREMENTS SPECIFICATION 36
3.1 Hardware Specification 36
3.2 Software Specification 36
3.3 Technology Specification 36
4 SYSTEM DESIGN 42
4.1 System Layout 42
4.2 Use Case Diagram 43
4.3 Activity Diagram 44
4.4 Class Diagram 45
4.5 Data Flow Diagram 46
4.6 Sequence Diagram 47
iv
5 IMPLEMENTATION 48
5.1 Project Implementation 48
5.2 Modules 51
5.3 Algorithm Description 55
6 TESTING 58
6.1 Testing 58
7 CONCLUSION AND FUTURE WORKS 61
7.1 Conclusion 61
7.2 Future works 62
8 APPENDIX -- A 65
APPENDIX -- B 72
REFERENCE 73
v
LIST OF FIGURES
vi
3.3 Numpy 38
3.4 HTML,CSS,JS 41
4.1 System Layout 42
4.2 Use Case Diagram 43
4.3 Activity Diagram 44
4.4 Class Diagram 45
4.5 Data Flow Diagram 46
4.6 Sequence Diagram 47
5.1 Camera visualization 51
5.2 Face detection 52
5.3 Extracting and classifying faces 53
5.4 Feature extraction 54
5.5 Emotion recognition 54
5.6 Music suggestion 55
5.7 HAAR feature 57
vii
LIST OF ABBREVISTIONS
UV Ultra Violet
RGB Red Green Blue
HSV Hue Saturation and Value
AI Artificial Intelligence
PET Positron Emission Tomography
fMRI functional Magnetic Resonance Imaging
HR Human Resources
API Application Program Interface
ATM Automated Teller Machine
SVM Support Vector Machine
ANN Artificial Neural Network
CNN Conventional Neural Network
LSTM Long Short-Term Memory
ELM Extreme Learning Machine
NLP Natural Language Processing
EEG Electroencephalography
ECG Electrocardiography
LBP Local Binary Pattern
GMM Gaussian Mixture Matrix
JAFFE Japanese Female Facial Expression
OpenCV Open Source Computer Vision
VGG Visual Geometry Group
viii
GPL General Public License
ONNX Open Neural Network Exchange
CUDA Compute Unified Device Architecture
GPU Graphics Processing Unit
GUI Graphical User Interface
WHATWG Web Hypertext Application Technology Working Group
SVG Scalable Vector Graphics
HTML Hyper Text Markup Language
CSS Cascading Style Sheet
DOM Document Object Model
ix
CHAPTER 1
1.INTRODUCTION
1.1 OBJECTIVE:
Facial expressions are a form of verbal communication where our project is
to introduce a system based on facial recognition. To identify the facial
expression and emotion of a person derived from live feed or static images. Using
OpenCV python library for facial object extraction and classification of
expressions expressed by the person. The FisherFaces algorithm can be used to
detect the face from the image through average classification. Data Sampling is
performed by dividing the data into Training and Testing data. The emotion can
be detected by taking various images of different emotions to train the data-set.
The model can be trained for improved accuracy and robust. The emotion of a
person can be analyzed and can be made stable through elements such as music.
Once the emotion is successfully detected with some basic error reductions, the
music can be played or suggested.
1
1.2 DEFINITION OF IMAGE:
An image is an artefact that depicts visual perception, such as a photograph or
other two-dimensional picture, particularly one that resembles a subject (usually
a physical object). It is a data representing a two-dimensional scene such as a
photograph or screen display, or three-dimensional, such as a statue or hologram,
that is composed of pixels arranged in a rectangular array. Each pixel may
consists of one or more bits of information representing the brightness of the
image including colour information encoded as RGB triples.
A volatile image is one that exists only for a short period of time. This may be
a reflection of an object by a mirror, a projection of a camera obscure, or a scene
displayed on a cathode ray tube. A fixed image, also called a hard copy, is one
that has been recorded on a material object, such as paper or textile by
photography or any other digital process. A mental image exists in an individual's
mind, as something one remembers or imagines. The subject of an image need
not be real; it may be an abstract concept, such as a graph, function, etc.
3
❖ Output in which result can be altered image or report that is based on image
analysis.
o Last, output the result of image processing. The result might be the
image changed by some way or it might be a report based on analysis
or result of the images.
In the 21st century image processing is usually "digital", but optical and
analog image methods are still used. Image processing is a method to perform
some operations on an image, in order to get an enhanced image or to extract
some useful information from it. It is a type of signal processing in which input
is an image and output may be image or characteristics/features associated with
that image. Nowadays, image processing is among rapidly growing technologies.
It forms core research area within engineering and computer science disciplines
too.
4
processing). The processing is applied on analog signals and it processes only
two-dimensional signals. The images are manipulated by electrical means by
varying the electrical signal. The common example include is the television
image. The images are manipulated by electrical signals. In analog image
processing, analog signals can be periodic or non-periodic.
Filtering
Digital filters are used to blur and sharpen digital images. Filtering can be
performed by:
5
The following examples show both methods:
Original
Image
Spatial
Lowpass
Spatial
Highpass
Pseudo-code:
6
Fourier
Lowpass
Fourier
Highpass
The analog image processing is applied The digital image processing is applied to
on analog signals and it processes only digital signals that work on analysing and
two-dimensional signals. manipulating the images.
Analog signal is time-varying signals so It improves the digital quality of the image
the images formed under analog image and intensity distribution is perfect in it.
processing get varied.
Analog image processing is a slower and Digital image processing is a cheaper and
costlier process. fast image storage and retrieval process.
Analog signal is a real-world but not good It uses good image compression techniques
quality of images. that reduce the amount of data required and
produce good quality of images
7
It is generally continuous and not broken It uses an image segmentation technique
into tiny components. which is used to detect discontinuity which
occurs due to a broken connection path.
Analog image’s pixel value must be Digital image’s pixel value must be
continuous. discrete.
The amplitude of analog image is infinite. The amplitude of digital image is finite.
It is quite impossible to store all the pixels It is quite possible to store all the pixels of
of analog image. digital image.
• Medical field
• Remote sensing
• Machine/Robot vision
• Color processing
• Pattern recognition
• Video processing
• Microscopic Imaging
• Others
8
Image sharpening and restoration
Image sharpening and restoration refers here to process images that have
been captured from the modern camera to make them a better image or to
manipulate those images in way to achieve desired result. It refers to do what
Photoshop usually does.
9
Medical field
The common applications of DIP in the field of medical is
• Gamma ray imaging
• PET scan
• X Ray Imaging
• Medical CT
• UV imaging
UV imaging
In the field of remote sensing , the area of the earth is scanned by a satellite
or from a very high ground and then it is analyzed to obtain information about
it. One particular application of digital image processing in the field of remote
sensing is to detect infrastructure damages caused by an earthquake.
10
The key steps include in the analysis are
• The extraction of edges
• Analysis and enhancement of various types of edges
Transmission and encoding
The very first image that has been transmitted over the wire was from
London to New York via a submarine cable. The picture that was sent is shown
below.
Machine/Robot vision
Apart form the many challenges that a robot face today , one of the biggest
challenge still is to increase the vision of the robot. Make robot able to see things
, identify them , identify the hurdles e.t.c. Much work has been contributed by
this field and a complete other field of computer vision has been introduced to
work on it.
11
Hurdle detection
Hurdle detection is one of the common task that has been done through
image processing, by identifying different type of objects in the image and then
calculating the distance between robot and hurdles.
12
Pattern recognition
Pattern recognition involves study from image processing and from
various other fields that includes machine learning ( a branch of artificial
intelligence). In pattern recognition , image processing is used for identifying the
objects in an images and then machine learning is used to train the system for
the change in pattern. Pattern recognition is used in computer aided diagnosis ,
recognition of handwriting , recognition of images ,etc.
Video processing
A video is nothing but just the very fast movement of pictures. The quality
of the video depends on the number of frames/pictures per minute and the quality
of each frame being used. Video processing involves noise reduction , detail
enhancement , motion detection , frame rate conversion , aspect ratio conversion
, color space conversion ,etc.
13
1.6 FACE DETECTION:
Face detection can be regarded as a specific case of object-class detection.
In object-class detection, the task is to find the locations and sizes of all objects
in an image that belong to a given class. Examples include upper torsos,
pedestrians, and cars. Face-detection algorithms focus on the detection of frontal
human faces. It is analogous to image detection in which the image of a person is
matched bit by bit. Image matches with the image stores in database. Any facial
feature changes in the database will invalidate the matching process. A reliable
face-detection approach based on the genetic algorithm and the eigen-
face technique:
Firstly, the possible human eye regions are detected by testing all the valley
regions in the gray-level image. Then the genetic algorithm is used to generate all
the possible face regions which include the eyebrows, the iris, the nostril and the
mouth corners.
14
launched a new selfie pay app called MasterCard Identity Check. Customers open
the app to confirm a payment using their camera, and that’s that. Facial
recognition is already used in store and at ATMs, but the next step is to do the
same for online payments. Chinese ecommerce firm Alibaba and affiliate
payment software Alipay are planning to apply the software to purchases made
over the Internet.
15
Fig 1.11 Face Detection for Access and Security
3. CRIMINAL IDENTIFICATION :
4. ADVERTISING :
The ability to collect and collate masses of personal data has given
marketers and advertisers the chance to get closer than ever to their target
16
markets. FaceTech could do much the same, by allowing companies to recognise
certain demographics – for instance, if the customer is a male between the ages
of 12 and 21, the screen might show an ad for the latest FIFA game. Grocery giant
Tesco plans to install OptimEyes screens at 450 petrol stations in the UK to
deliver targeted ads to customers. According to company CEO Simon Sugar, the
cameras could change the face of British retail. Perhaps he’s right – but only if
the cameras can correctly identify customers. Being classified as the wrong age
or gender is far less amusing than having your name spelt wrong on a Starbucks
cup.
5. HEALTHCARE :
17
identity theft. Of course, FaceTech is by no means fool proof. Gaining access to
possessions using physical traits could even be counterintuitive for security. A
face, as social robots like Nadine have shown us, is easily replicated. And when
it comes to public adoption, some people are reluctant to switch to contactless
cards, let alone abandon them completely. For the most part, though, facial
recognition technology seems to be encouraging a more seamless relationship
between people, payments and possessions.
1.7 EMOTION :
Emotions are biological states associated with the nervous system brought
on by neurophysiological changes variously associated with thoughts, feelings,
behavioural responses, and a degree of pleasure or displeasure. There is currently
no scientific consensus on a definition. Emotion is often intertwined
with mood, temperament, personality, disposition, creativity and motivation.
Research on emotion has increased significantly over the past two decades with
many fields contributing including psychology, neuroscience, affective,
neuroscience, endocrinology, medicine, history, sociology of emotions,
and computer science. The numerous theories that attempt to explain the origin,
neurobiology, experience, and function of emotions have only fostered more
intense research on this topic. Current areas of research in the concept of emotion
include the development of materials that stimulate and elicit emotion. In
addition PET scans and fMRI scans help study the affective picture processes in
the brain.
18
human emotions. Using facial emotion detection smart cars can alert the driver
when he is feeling drowsy.
19
Fig 1.15 Emotion Detection in Interviews
20
Fig 1.16 Emotion Detection in Video Gaming
Such product feedback can be taken by analyzing a live feed of the user
and detecting his facial emotions. While feelings of frustration and anger are
commonly experienced in advanced video games, making use of facial emotion
detection will help understand which emotions are experienced at what points in
the game. It is also possible that some unexpected or undesirable emotions are
observed during the game. Taking feedback from the user has experienced the
game can be inefficient. This is because it can often be difficult to put an
experience into words. Moreover, users may be unable to remember what exactly
they went through emotionally across different parts of the game. Facial emotion
detection is a practical means of going beyond the spoken or written feedback
and appreciating what the user is experiencing. When feedback is taken in this
format, it becomes genuinely non-intrusive when it comes to user experience. At
the same time, such feedback is more reliable than other forms.
4. MARKET RESEARCH :
Traditional market research companies have employed verbal methods
mostly in the form of surveys to find the consumers wants and needs. However,
such methods assume that consumers can formulate their preferences verbally
and the stated preferences correspond to future actions which may not always be
right.
21
Fig 1.17 Facial Recognition Market
22
CHAPTER 2
2. SYSTEM ANALYSIS
2.1 Existing Solution:
This paper discusses the application of feature extraction of facial
expressions with combination of neural network for the recognition of
different facial emotions (happy, sad, angry, fear, surprised, neutral etc..).
Humans are capable of producing thousands of facial actions during
communication that vary in complexity, intensity, and meaning. In Emotion
recognition using brain activity the developer Robert Horlings has used brain
activities which is toughest task to do as it become expensive, complex and
also time consuming when we try to measure human brain with
Electroencephalography (eeg).
2.1.1 Approaches:
The accuracy of emotion recognition is usually improved when it combines
the analysis of human expressions from multimodal forms such as texts,
physiology, audio, or video. Different emotion types are detected through the
integration of information from facial expressions, body movement and gestures,
and speech. The technology is said to contribute in the emergence of the so-called
emotional or emotive Internet.
24
reasonable classification accuracy compared to other approaches, but one of the
challenges in achieving good results in the classification process, is the need to
have a sufficiently large training set.
25
2.1.5 Datasets:
Data is an integral part of the existing approaches in emotion recognition and
in most cases it is a challenge to obtain annotated data that is necessary to
train machine learning algorithms. For the task of classifying
different emotion types from multimodal sources in the form of texts, audio,
videos or physiological signals, the following datasets are available:
1. HUMAINE: provides natural clips with emotion words and context labels
in multiple modalities.
2. Belfast database: provides clips with a wide range of emotions from TV
programs and interview recordings.
3. SEMAINE: provides audiovisual recordings between a person and
a virtual agent and contains emotion annotations such as angry, happy,
fear, disgust, sadness, contempt, and amusement.
4. IEMOCAP: provides recordings of dyadic sessions between actors and
contains emotion annotations such as happiness, anger, sadness,
frustration, and neutral state.
5. eNTERFACE: provides audiovisual recordings of subjects from seven
nationalities and contains emotion annotations such as happiness, anger,
sadness, surprise, disgust, and fear.
6. DEAP:provides electroencephalography (EEG), electrocardiography (EC
G), and face video recordings, as well as emotion annotations in terms
of valence, arousal, and dominance of people watching film clips.
7. DREAMER: provides electroencephalography (EEG)
and electrocardiography (ECG) recordings, as well
as emotion annotations in terms of valence, arousal, and dominance of
people watching film clips.
8. MELD: is a multiparty conversational dataset where each utterance is
labeled with emotion and sentiment. MELD provides conversations in
26
video format and hence suitable for multimodal emotion recognition
and sentiment analysis. MELD is useful for multimodal sentiment
analysis and emotion recognition, dialogue systems and emotion
recognition in conversations.
Description:
In this paper, it proposes a high-performance emotion recognition system
for mobile applications. The embedded camera in a smart phone captures the
video of the user. The Bandlet transform is applied to some selective frames to
reduce the burden of processing to give some subband images. Local binary
patterns (LBP) histogram is calculated from the subband images which is a
powerful gray-level image descriptor which is used to describe the features of the
frames. A Gaussian mixture model (GMM) based classifier is used as a classifier
for training and testing. The proposed emotion recognition system is evaluated
using several databases for improved accuracy. An emotion recognition system
for mobile applications has been proposed. The emotions are recognized by face
images. The Bandlet transform and the LBP are used as features, which are then
selected by the KW feature selection method. The GMM based classifier is
applied to recognize the emotions. Two publicly available databases are used to
validate the system. The proposed system achieved 99.8% accuracy using the
JAFFE database, and 99.7% accuracy using the CK database. It takes less than
1.4 seconds to recognize one instance of emotion. The high performance and the
less time requirement of the system make it suitable to any emotion-aware mobile
27
applications. In a future study, we want to extend this work to incorporate
different input modalities of emotion.
In the proposed system, facial video is captured by an embedded camera
of a smart phone. Some representative frames are extracted from the video, and a
face detection module is applied to extract the face regions in the frames. The
Bandlet transform is realized on the face regions, and the resultant subband is
divided into non-overlapping blocks. Local binary patterns’ histograms are
calculated for each block, and then are concatenated over all the blocks. The
Kruskal–Wallis feature selection is applied to select the most dominant bins of
the concatenated histograms. The dominant bins are then fed into a Gaussian
mixture model-based classifier to classify the emotion. Experimental results show
that the proposed system achieves high recognition accuracy in a reasonable time.
Advantage:
• Provides an emotion recognition system for mobile application.
• It achieves an accuracy of about 99.8%.
• It takes less than 1.4 seconds to recognize one instance of emotion.
• Using histograms, only the representative frames are selected to reduce
the burden of processing.
Description:
In the field of education, existing virtual learning environments with
success simulate interaction at a psychological feature level throughout ancient
teaching processes. Students show completely different expressions in videos,
lecturers will determine whether or not the scholars perceive their teaching
28
content in line with student’s with completely different expressions, and might
modify their teaching programs. In virtual environments, student’s emotional
knowledge is obtained from a series of video frames. Therefore, to achieve real
time understanding of student’s emotions, accuracy and time must be balanced.
To improve accuracy, a lot of video frames square measure needed, increasing
computation time. This paper proposes a learning emotion recognition model,
which consists of three stages: Feature extraction, subset feature and emotion
classifier. A Haar Cascades method is used to detect the input image, a face, as
the basis for the extraction of eyes and mouth, and then through the Sobel edge
detection to obtain the characteristic value. Through Neural Network classifier
training, six kinds of different emotional categories are obtained. Experiments
using JAFFE database show that the proposed method has high classification
performance. Experimental results show that the model proposed in this paper is
consistent with the expressions from the learning situation of students in virtual
learning environments. This paper demonstrates that emotion recognition based
on facial expressions is feasible in distance education, permitting identification
of a student’s learning status in real time. Therefore, it can help teachers to change
teaching strategies in virtual learning environments according to the student’s
emotions.
Advantage:
• Solves the problems of emotion recognition based on facial recognition in
virtual learning environments.
• It can be applied for real distance education.
• Optimization of recognition of emotional changes in the facial features of
online learners.
Drawback:
• It does not involve the illumination and pose of the image, it is uncertain
how much these factors influence facial expressions.
• The accuracy of recognition of faces is comparatively less.
29
3.Emotion Detection using Image Processing in Python
Author: Raghav Puri, Archit Gupta, Manas Sikri, Mr. Mohit Tiwari, Dr. Nitish
Pathak, Dr. Shivendra Goel.
Description:
It is a very efficient way of image processing through which an image can
be converted into its digital form subsequently performing various operations on
it. This is a technique similar to signal processing, in which the input given is a
2D image, which is a collection of numbers ranging from 0 to 255 which denotes
the corresponding pixel value. The image is converted into digital form for good
resolution and high quantization level for human perception of different shades.
The subject on each image is analyzed, converted to grayscale, cropped and saved
to a directory which can be compiled as training set and classification set. The
user’s emotion using its facial expressions will be detected. These expressions
can be derived from the live feed via system’s camera or any pre-existing image
available in the memory. Emotions possessed by humans can be recognized and
has a vast scope of study in the computer vision industry upon which several
researches have already been done. The work has been implemented using Python
(2.7), Open Source Computer Vision Library (OpenCV) and NumPy. The
scanned image (testing dataset) is being compared to training dataset and thus
emotion is predicted. The objective of this paper is to develop a system which can
analyze the image and predict the expression of the person. The study proves that
this procedure is workable and produces valid results.
Advantage:
• Artificial Intelligence is used to solve the intriguing tasks in emotion
detection.
• The image is converted into digital form to enhance for good resolution
and high quantization level.
• The input is a 2D image which consists of pixel values.
30
Drawback:
• The system detects only less than three emotions.
• The optimum accuracy is nearly 83% which is relatively less.
• The properties of the image can be affected due to converting it into
digital form.
Description:
Face detection is the most popular area of research in the vision of
computer science. It is a computer technology which is being used in a variety of
applications that identifies human faces in digital images. The research under this
field is expanding in many areas of science such as psychology. Face detection is
one of the most talked about in technology. Localization of human faces is
considered as the primary and the initial stage in study of face detection . The
application is used for tracking and detecting faces in videos or still images
through cameras which can be used for multipurpose activities. The intention of
the paper is to deep study of face detection using OpenCV. A tabular comparison
is provided to understand the available algorithms in an easier manner. It talks
about various types of algorithms that exist like Adaboost, Haar cascades. The
paper aims to help in understanding the best prerequisites for face detection. Face
detection and tracking is used in many of the fields these days. It is collaborated
in every section of the trade and some are effectively and efficiently using it,
others still depend the on the old methods.
31
Advantage:
• It compares all the algorithm in terms of space and time paradigm.
• It talks about the difference in face detection via performing it with
OpenCV and performing it with matlab.
• OpenCV is fast, portable and free of cost.
• It provides an understanding of the best prerequisites for face detection.
Drawback:
• The feature extraction using HAAR Cascades is time consuming.
• It cannot process without the variable light source for face detection
through camera.
Description:
Facial expressions are a form of nonverbal communication. Various studies
have been done for the classification of these facial expressions. There is strong
evidence for the universal facial expressions of eight emotions which include:
neutral happy, sadness, anger, contempt, disgust, fear, and surprise. So it is very
important to detect these emotions on the face as it has wide applications in the
field of Computer Vision and Artificial Intelligence. These fields are researching
on the facial emotions to get the sentiments of the humans automatically. In
Robotics, emotions classification can be used to enhance human-robot
interactions since the robot is capable of interpreting a human reaction . In this
project, the images datasets were organized properly and then sorted accordingly.
Then the face was detected in all the images using the Haar filters in OpenCV as
it has few face recognized classes and then detect, crop and save faces. The faces
32
were detected in the frames according to the facial landmarks which will contain
the eyes, eyebrows, nose, mouth, corners of the face. The classification is done
using the supervised learning (SVM) in this project as it gave better accuracy.
The training and classification sets were created and we randomly sample and
train on 80% of the data and classify the remaining 20%. This whole work was
done on the Cohn-Kanade datasets of static images.
Advantage:
• Provides a fully automatic recognition of facial emotions using the
computer vision and machine learning algorithms.
• Provides an accuracy of 94.1%.
• The system can be extended to the deep learning system using the
Convolutional Neural networks with an accuracy of 99.5%.
• Each image sequence is processed for the development of an emotional
expression.
Drawback:
• The classifier needs to be trained in order to achieve accuracy in
extraction of emotion.
• Using SVM, it contained a little errors in the prediction which reduced
the accuracy of processing.
Description:
Emotions often mediate and facilitate interactions among human beings.
Thus, understanding emotion often brings context to seemingly bizarre and/or
complex social communication. Emotion can be recognized through a variety of
means such as voice intonation, body language, and more complex methods such
33
electroencephalography (EEG). However, the easier, more practical method is to
examine facial expressions. There are seven types of human emotions shown to
be universally recognizable across different cultures: anger, disgust, fear,
happiness, sadness, surprise, contempt. Interestingly, even for complex
expressions where a mixture of emotions could be used as descriptors, cross-
cultural agreement is still observed. The task of emotion recognition is
particularly difficult for two reasons: 1) There does not exist a large database of
training images and 2) classifying emotion can be difficult depending on whether
the input image is static or a transition frame into a facial expression. The latter
issue is particularly difficult for real-time detection where facial expressions vary
dynamically. The goal of this project was to implement real-time facial emotion
recognition. Using the custom trained VGG network with a face-detector
provided by OpenCV, we successfully implemented an application wherein an
emoji indicating one of six expressions (anger, fear, neutral, happy, sad, surprise)
is superimposed over a user’s face in real time. First, a much larger dataset should
be designed to improve the model’s generality. While it achieved > 90% accuracy
in laboratory conditions (perfect lighting, camera at eye level, subject facing
camera with an exaggerated expression), any deviation from that caused the
accuracy to fall significantly. In particular, any shadow on a subject’s face would
cause an incorrect classification of ‘angry’. Also, fully training a network other
than VGG might yield substantial improvements in computation speed, since
VGG is relatively slow.
Advantages:
• It is able to classify a subject’s emotion reliably at higher frame-rates.
• It achieves higher accuracy in laboratory conditions.
• The emotion detected by the CNN is indicated by the type of emoji
superimposed over the subject’s face.
34
2.3 Proposed Solution
Humans have always had the innate ability to recognize and distinguish
between faces from their emotions where now computers are able to do the same.
This opens up tons of applications. We have developed a system which can detect
about all the expressed emotions using OpenCV. The inbuilt camera that is
present in laptop, mobile devices, etc is accessed to capture the image of the
person. The image can be captured from live-feed or static images from the
camera. The face from the webcam is detected and pre-process the image of the
face using a pre-trained HAAR classifier supplied with OpenCV. The
FisherFaces algorithm is used to detect the face from the processed image by
converting the image into grayscale through average classification. In that sense,
an individual's components do not dominate over the others. The emotion is
detected by taking varied images of different emotions to train the data-set which
can be used to classify emotions expressed based on differences in patterns.
Humans are used to taking in non-verbal cues from facial emotions. This system
can detect four different human emotions - Happy, Sad, Neutral, Angry. The
trained model is capable to detect all the mentioned emotions in real-time and can
be trained for improved accuracy and robust. Based on the emotion extracted with
some basic error reductions, music is suggested to enhance the state of the person
through various genres of music. The system can auto-play music or suggest
preference of music from the infrastructure available on the target computers on
which the software will be run.
35
CHAPTER 3
3. REQUIREMENT SPECIFICATION
❖ CPU(Desktop)
➢ Windows – x86 and 64 bit architecture
➢ Mac – x64 bit architecture
➢ Linux – x64 bit architecture
❖ RAM
➢ Desktop – Minimum 4GB
❖ Storage
➢ Desktop – Minimum 512 MB
❖ Camera
➢ 1.5 megapixel or higher
❖ Operating System(OS)
➢ Windows - Windows 7 and Windows 10
➢ MacOS - Sierra 10.12.6+
➢ Linux – Ubuntu 16.04, Ubuntu 18.04 and CentOS 7
❖ Python
➢ Version – 3.6
36
often described as a "batteries included" language due to its
comprehensive standard library. Python interpreters are available for
many operating systems. A global community of programmers develops and
maintains CPython, an open source reference implementation. A non-profit
organization, the Python Software Foundation, manages and directs resources for
Python and CPython development.
37
Fig 3.2 OpenCV
❖ 3.3.3 Numpy-1.18.1
NumPy is a library for the Python programming language, adding support
for large, multi-dimensional arrays and matrices, along with a large collection
of high-level mathematical functions to operate on these arrays. NumPy targets
the CPython reference implementation of Python, which is a non-
optimizing bytecode interpreter. Mathematical algorithms written for this
version of Python often run much slower than compiled equivalents. NumPy
addresses the slowness problem partly by providing multidimensional arrays and
functions and operators that operate efficiently on arrays, requiring rewriting
some code, mostly inner loops using NumPy. Array in Numpy is a table of
elements (usually numbers), all of the same type, indexed by a tuple of positive
integers. In Numpy, number of dimensions of the array is called rank of the
array.A tuple of integers giving the size of the array along each dimension is
known as shape of the array. An array class in Numpy is called as ndarray.
Elements in Numpy arrays are accessed by using square brackets and can be
initialized by using nested Python Lists.
38
❖ 3.3.4 Eel-0.12.2
There are several options for making GUI apps in Python, but if you want
to use HTML/JS (in order to use jQueryUI or Bootstrap, for example) then
you generally have to write a lot of boilerplate code to communicate from the
Client (Javascript) side to the Server (Python) side.The closest Python
equivalent to Electron (to my knowledge) is cefpython. It is a bit heavy weight
for what I wanted. Eel is not as fully-fledged as Electron or cefpython - it is
probably not suitable for making full blown applications like Atom - but it is
very suitable for making the GUI equivalent of little utility scripts that you
use internally in your team. For some reason many of the best-in-class number
crunching and maths libraries are in Python (Tensorflow, Numpy, Scipy etc)
but many of the best visualization libraries are in Javascript (D3, THREE.js
etc). Hopefully Eel makes it easy to combine these into simple utility apps for
assisting your development. Install through pip : pip install eel
An Eel application will be split into a frontend consisting of various web-
technology files (.html, .js, .css) and a backend consisting of various Python
scripts.
❖ 3.3.5 HTML5
HTML5 is a markup language used for structuring and presenting content
on the World Wide Web. HTML5 was the fifth and last major version
of HTML that is a World Wide Web Consortium (W3C) recommendation.
The current specification is known as the HTML Living Standard and is
maintained by a consortium of the major browser vendors
(Apple, Google, Mozilla, and Microsoft), the Web Hypertext Application
Technology Working Group (WHATWG). HTML5 includes detailed
processing models to encourage more interoperable implementations; it
extends, improves and rationalizes the markup available for documents, and
introduces markup and application programming interfaces (APIs) for
39
complex web applications.[6] For the same reasons, HTML5 is also a
candidate for cross-platform mobile applications, because it includes features
designed with low-powered devices in mind. Many new syntactic features are
included. To natively include and handle multimedia and graphical content,
the new <video> , <audio> and <canvas> elements were added, and support
for scalable vector graphics (SVG) content and MathML for mathematical
formulas. To enrich the semantic content of documents, new page structure
elements such as <main>, <section>, <article>, <header>, <footer>, <aside>,
<nav>, and <figure> are added. New attributes are introduced, some elements
and attributes have been removed, and others such as <a>, <cite>, and
<menu> have been changed, redefined, or standardized.
❖ 3.3.6 CSS3
Cascading Style Sheets (CSS) is a style sheet language used for
describing the presentation of a document written in a markup
language like HTML. CSS is a cornerstone technology of the World Wide
Web, alongside HTML and JavaScript. CSS is designed to enable the
separation of presentation and content, including layout, colors,
and fonts. This separation can improve content accessibility, provide more
flexibility and control in the specification of presentation characteristics,
enable multiple web pages to share formatting by specifying the relevant CSS
in a separate .css file, and reduce complexity and repetition in the structural
content. Separation of formatting and content also makes it feasible to present
the same markup page in different styles for different rendering methods, such
as on-screen, in print, by voice (via speech-based browser or screen reader),
and on Braille-based tactile devices. CSS also has rules for alternate
formatting if the content is accessed on a mobile device.
40
❖ 3.3.7 JavaScript
JavaScript, often abbreviated as JS, is a programming language that
conforms to the ECMAScript specification. JavaScript is high-level,
often just-in-time compiled, and multi-paradigm. It has curly-bracket
syntax, dynamic typing, prototype-based object-orientation, and first-class
functions. Alongside HTML and CSS, JavaScript is one of the core
technologies of the World Wide Web. JavaScript enables interactive web
pages and is an essential part of web applications. The vast majority
of websites use it for client-side page behavior and all major web
browsers have a dedicated JavaScript engine to execute it. As a multi-
paradigm language, JavaScript supports event-driven, functional,
and imperative programming styles. It has application programming
interfaces (APIs) for working with text, dates, regular expressions,
standard data structures, and the Document Object Model (DOM). However,
the language itself does not include any input/output (I/O), such
as networking, storage, or graphics facilities, as the host environment
(usually a web browser) provides those APIs. JavaScript engines were
originally used only in web browsers, but they are now embedded in server-
side website deployments, usually via Node.js. They are also embedded in a
variety of applications created with frameworks such
as Electron and Cordova. Although there are similarities between JavaScript
and Java, including language name, syntax, and respective standard libraries,
the two languages are distinct and differ greatly in design.
4. SYSTEM DESIGN
42
4.2 Use Case Diagram
43
4.3 Activity Diagram
44
4.4 Class Diagram
45
4.5 Data Flow Diagram
46
4.6 Sequence Diagram
47
CHAPTER 5
5. IMPLEMENTATION
5.1 Project Implementation
Facial Detection — Ability to detect the location of face in any input image
or frame. The output is the bounding box coordinates of the detected faces
Emotion Detection — Classifying the emotion on the face as happy, angry,
sad, neutral, surprise, disgust or fear
Humans have always had the innate ability to recognize and distinguish
between faces from their emotions where now computers are able to do the same.
This opens up tons of applications. We have developed a system which can detect
about all the expressed emotions using OpenCV. The inbuilt camera that is
present in laptop, mobile devices, etc is accessed to capture the image of the
person. The image can be captured from live-feed or static images from the
camera. The face from the webcam is detected and pre-process the image of the
face using a pre-trained HAAR classifier supplied with OpenCV. The
FisherFaces algorithm is used to detect the face from the processed image by
converting the image into grayscale through average classification. In that sense,
an individual's components do not dominate over the others. The emotion is
detected by taking varied images of different emotions to train the data-set which
can be used to classify emotions expressed based on differences in patterns.
Humans are used to taking in non-verbal cues from facial emotions. This system
can detect four different human emotions - Happy, Sad, Neutral, Angry. The
trained model is capable to detect all the mentioned emotions in real-time and
can be trained for improved accuracy and robust. Based on the emotion extracted
with some basic error reductions, music is suggested to enhance the state of the
person through various genres of music. The system can auto-play music or
suggest preference of music from the infrastructure available on the target
computers on which the software will be run.
48
❖ Image Processing:
It consists of three basic steps :
1.) Scanning the image: a raw image is acquired which has to be
processed. It can be expressed in form of pixels as stated above.
The aim of this step is to extract information which is suitable for
computing.
2.) Processing and Enhancing it: -the image is converted into digital
form by using a digitizer which samples and quantizes the input
signals. The rate of sampling should be high for good resolution
and high quantization level for human perception of different
shades using different using gray-scale.
3.) The obtained result describes the property of the image and further
classifies the image.
❖ Facial detection is the first part of our pipeline. We have used the python
library Face Detection that we found easy to install and very accurate in detecting
faces. This library scans the input image and returns the bounding box coordinates
of all detected faces. The FisherFace algorithm of the python Face Detection
library is used to detect the face present in an image by using a classifier to obtain
the feature points of the image to detect and obtain the face accurately with the
objects present.
❖ Face recognition involves three basic steps which include face detection,
face recognition, and face extraction. Any system needs to encapsulate the image
and then manage as well as record the vital features to determine the location of
the face. For the recognition of the captured image it keeps records of various
features such as skin color, skin tone etc. Face recognition takes an image from
49
a video or a camera as input and outputs the identified image topic. Facial
features may include regions in the face, variations in the face structure, face cuts
and angles that have been formatted and styled. Face extraction includes
grabbing of the features from camera. Face detection includes removal of the
background and focusing on the foreground eliminating any other elements apart
from the face region.
❖ The classifier will work best if the training and classification images are
all of the same size and have (almost) only a face on them (no clutter). We need
to find the face on each image, convert to grayscale, crop it and save the image
to the dataset. We can use a HAAR filter from OpenCV to automate face finding.
Actually, OpenCV provides 4 pre-trained classifiers, so to be sure we detect as
many faces as possible let’s use all of them in sequence, and abort the face search
once we have found one. Get them from the OpenCV directory or from here and
extract to the same file you have your python files.
Create another folder called “dataset”, and in it create subfolders for each
emotion (“neutral”, “anger”, etc.).
❖ The dataset has been organised and is ready to be recognized, but first we
need to actually teach the classifier what certain emotions look like. The usual
approach is to split the complete dataset into a training set and a classification
set. We use the training set to teach the classifier to recognize the to-be-predicted
labels, and use the classification set to estimate the classifier performance.
50
5.2 Modules
5.2.1. Camera Visualization and Capture:
There is a rapidly growing demand for using smart cameras for various
biometric applications in surveillance. Although having a small form-factor,
most of these applications demand huge processing performance for real-time
processing. Face recognition is one of those applications. In this paper we show
that we can run face detection in real-time by implementing the algorithm for
face detection and capture from the image obtained. The inbuilt camera that is
present in laptop, mobile devices, etc. is accessed to capture the image of the
person. The image can be captured from live-feed or static images by enabling
the camera.
52
Fig 5.3 Extracting and classifying faces
53
Fig 5.4 Feature extraction
5.2.5. Emotion Recognition:
The dataset which has been organized and is ready to be recognized, but
initially, we need to actually train the classifier what particular emotions look
like. The approach we have used is to split the complete dataset into a training
set and a classification set. We use the training set to teach the classifier to
recognize the labels to be predicted, and used the classification set to estimate
the performance of the classifier. After creating the training and classification
set, we randomly sample and train on 80% of the data and classify the remaining
20%, and repeat the process 20 times. After training the fisher face classifier the
emotions were predicted.
54
5.2.6 Music Suggestion:
Music plays a very important role in human's daily life and in the modern
advanced technologies. Usually, the user has to face the task of manually
browsing through the playlist of songs to select. Here we are proposing an
efficient and accurate model, that would generate a playlist based on current
emotional state and behaviour of the user. Existing methods for automating the
playlist generation process are computationally slow, less accurate and
sometimes even require use of additional hardware like EEG or sensors. Speech
is the most ancient and natural way of expressing feelings, emotions and mood
and its and its processing requires high computational, time, and cost. This
proposed system based on real-time extraction of facial expressions to classify
into a specific emotion that will generate a playlist automatically such that the
computation cost is relatively low.
56
classifier. Then we need to extract features from it. For this, haar features of fig
5.1 are used. They are just like our convolutional kernel. Each feature is a single
value obtained by subtracting sum of pixels under white rectangle from sum of
pixels under black rectangle.
Now all possible sizes and locations of each kernel is used to calculate
plenty of features. For each feature calculation, we need to find sum of pixels
under white and black rectangles. To solve this, they introduced the integral
images. It simplifies calculation of sum of pixels, how large may be the number
of pixels, to an operation involving just four pixels.
For each feature, it finds the best threshold which will classify the faces to
positive and negative. But obviously, there will be errors or misclassifications.
We select the features with minimum error rate, which means they are the
features that best classifies the face and non-face images. After each
classification, weights of misclassified images are increased. Then again same
process is done. New error rates are calculated. Also new weights. The process
is continued until required accuracy or error rate is achieved or required number
of features are found.
57
CHAPTER 6
6. TESTING
6.1 Testing
In this project, different types of testing in different environments which are
given as follows:
6.1.1 Alpha Testing
• It is the most common type of testing used in the Software industry. The
objective of this testing is to identify all possible issues or defects before
releasing it into the market or to the user.
• Alpha Testing is carried out at the end of the software development phase
but before the Beta Testing. Still, minor design changes may be made as a
result of such testing.
• Alpha Testing is conducted at the developer’s site. In-house virtual user
environment can be created for this type of testing.
58
• So end-user actually uses the software and shares the feedback to the
company. Company then takes necessary action before releasing the
software to the worldwide.
• Internal system design is not considered in this type of testing. Tests are
based on the requirements and functionality
• BLACK BOX TESTING is defined as a testing technique in which
functionality of the Application Under Test (AUT) is tested without
looking at the internal code structure, implementation details and
knowledge of internal paths of the software. This type of testing is based
entirely on software requirements and specifications. In Black Box Testing
we just focus on inputs and output of the software system without bothering
about internal knowledge of the software.
59
6.1.5 Integration Testing
• This term is often used interchangeably with ‘stress' and ‘load' testing.
Performance Testing is done to check whether the system meets the
performance requirements. Different performance and load tools are used
to do this testing.
60
CHAPTER 7
61
7.2 Future Works
In this project, we have presented the fully automatic recognition of facial
emotions using the computer vision and machine learning algorithms which
classify these eight different emotions. We tried many algorithms for the
classification but the best which came out of the results was the support vectors
machines with the accuracy of around 94.1%. Our results imply that user
independent, fully automatic real-time coding of facial expressions in the
continuous video stream is an achievable goal with present power of the
computer, at least for applications in which frontal views can be assumed using
the webcam. This machine learning based system for the emotion recognition can
be extended to the deep learning system using the Convolutional Neural networks
which will have many layers and the chances of getting much higher accuracy is
there around 99.5%. This project can be extended in which it will detect as many
emotions of different people in one frame in the real-time videos. Emotion
recognition is going to be very useful in the near future in the research field of
robotics and artificial Intelligence for example if a robot can sense the sentiment
of any human and that robot can act accordingly without any intervention of any
other humans. This automatic machine learning system for emotion recognition
can also be extended with the detection of mixed emotions other than these eight
universal emotions.
62
software products depends hugely upon human emotions. A products overall look
and feel can also alter human feelings which in turn makes the person buy the
product or not. Thus, researching about different emotional states of a human
body and how it is influenced by usage of different products is a matter of prime
importance for related industry professionals.
❖ Immersive gaming
Video games constitute a large chunk of entertainment industry. Thus, in
order to make these games much more intensive and intrusive, video game
creators base their research on different types of human emotions commonly
found. In order to allure more and more players, video games are made in such a
way that they incorporate human emotions naturally into their game play.
63
❖ Safe Driving
Being able to recognize those actions for drivers or passengers can help
smart cars to better understand what’s going on inside the car. Then based on
those observations, control can pass between the smart car and driver - or triggers
can be put in place, where an alarm goes off to the driver if he becomes drowsy
before he drifts from his lane. These features not only can enhance the road safety
but it will enhance the driving experience of people for next generations of family
cars and public transportation.
❖ Social Media
Social media like Twitter and Facebook is full of emotions, feelings and
opinions of people all over the world. However, analysing and classifying text on
the basis of emotions is a big challenge and can be considered as an advanced
form of Sentiment Analysis. Analysing the emotion through camera images is
also a big challenge and can be expanded for future enhancement in detecting
emotions while texting.
64
APPENDIX
Capture.py
def grab_face():
ret, frame = video_capture.read()
cv2.imshow("Video", frame)
cv2.imwrite('test.jpg', frame)
cv2.waitKey()
cv2.imwrite("images/main%s.jpg" %count, frame)
gray = cv2.imread('test.jpg',0)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
clahe_image = clahe.apply(gray)
return clahe_image
def detect_face():
clahe_image=grab_face()
face = facecascade.detectMultiScale(clahe_image, scaleFactor=1.1, minNeigh
bors=15, minSize=(10, 10), flags=cv2.CASCADE_SCALE_IMAGE)
if len(face) >= 1:
faceslice=crop(clahe_image, face)
#return faceslice
else:
print("No/Multiple faces detected!!, passing over the frame")
def save_face(emotion):
print("\n\nLook "+emotion+" untill the timer expires and keep the same emo
tion for some time.")
#winsound.Beep(frequency, duration)
print('\a')
while len(facedict.keys())<16:
detect_face()
65
for i in facedict.keys():
path, dirs, files = next(os.walk("dataset/%s" %emotion))
file_count = len(files)+1
cv2.imwrite("dataset/%s/%s.jpg" %(emotion, (file_count)), facedict[i])
facedict.clear()
def update_model(emotions):
print("Update mode for model is ready")
checkForFolders(emotions)
for i in facedict.keys():
pred, conf = fishface.predict(facedict[i])
cv2.imwrite("images/%s.jpg" %i, facedict[i])
prediction.append(pred)
confidence.append(conf)
output = emotions[max(set(prediction), key=prediction.count)]
print("You seem to be %s" %output)
facedict.clear()
return output;
Display.py
def show_frame():
_, frame = cap.read()
frame = cv2.flip(frame, 1)
cv2image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA)
img = Image.fromarray(cv2image)
imgtk = ImageTk.PhotoImage(image=img)
lmain.imgtk = imgtk
lmain.configure(image=imgtk)
lmain.after(10, show_frame)
66
Update_Model.py
def update(emotions):
run_recognizer(emotions)
print("Saving model...")
fishface.save("model.xml")
print("Model saved!!")
def make_sets(emotions):
training_data=[]
training_label=[]
def run_recognizer(emotions):
training_data, training_label=make_sets(emotions)
print("Training model...")
print("The size of the dataset is "+str(len(training_data))+" images")
fishface.train(training_data, np.asarray(training_label))
model.xml
<?xml version="1.0"?>
<opencv_storage>
<opencv_fisherfaces>
<threshold>1.7976931348623157e+308</threshold>
<num_components>3</num_components>
<mean type_id="opencv-matrix">
<rows>1</rows>
<cols>122500</cols>
<dt>d</dt>
<data>
1.2516740088105728e+02 1.2442511013215861e+02
1.2354405286343614e+02 1.2265859030837005e+02
1.2179735682819384e+02 1.2093171806167402e+02
1.1991850220264318e+02 1.1878414096916300e+02
1.1750881057268724e+02 1.1566960352422909e+02
1.1364977973568283e+02 1.1172246696035243e+02
1.0980176211453745e+02 1.0812555066079297e+02
1.0634801762114539e+02 1.0465638766519825e+02
67
1.0307709251101322e+02 1.0179074889867843e+02
1.0057929515418503e+02 9.9590308370044056e+01
9.8638766519823804e+01 9.7905286343612346e+01
9.7017621145374463e+01 9.6066079295154196e+01
</data>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4271991251107886e+02 -3.3896220482578474e+02
-3.6573086542596559e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4273866263198272e+02 -3.3899534241133790e+02
-3.6619194488891287e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4273509221926548e+02 -3.3891537879463488e+02
-3.6688125336462372e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4274420341218968e+02 -3.3964100091790311e+02
-3.6714775703502420e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4275364862484702e+02 -3.3999607975979859e+02
-3.6699728036397511e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4279189729893000e+02 -3.4136939981878021e+02
-3.6957102621035199e+02</data></_>
<_ type_id="opencv-matrix">
68
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4276148305262836e+02 -3.3933532551921923e+02
-3.6923730039222505e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4275579744005893e+02 -3.3982389303019039e+02
-3.6665379210691754e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4276253030710495e+02 -3.4151185469706076e+02
-3.6853576120018147e+02</data></_></projections>
<labels type_id="opencv-matrix">
<rows>454</rows>
<cols>1</cols>
<dt>i</dt>
<data>
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3</data></labels>
<labelsInfo>
</labelsInfo></opencv_fisherfaces>
</opencv_storage>
69
Code.js
function addq(elem) {
console.log(elem.id);
var x = elem.id.charAt(1);
if (!songrun) {
var z = songs[x][0];
document.getElementById("sname").innerHTML = sname[x];
document.getElementById("sel").src = z;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + son
gs[x][3] + "')";
songrun = true;
return;
}
if (bool[x] == true)
return;
bool[x] = true;
var l = document.createElement("label");
l.id = "e" + eqc;
l.name = x;
l.innerHTML = sname[x] + "<br>";
//var text=document.createTextNode(sname[x]+"<br>");
//l.appendChild(text);
document.getElementById("queue").appendChild(l);
eqc = eqc + 1;
}
function nextsong() {
if (sqc == eqc) {
alert("Queue is empty.");
return;
}
var elem = document.getElementById("e" + sqc);
var xa = elem.name;
var pa = songs[xa][0];
bool[xa] = false;
document.getElementById("sname").innerHTML = sname[xa];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[x
a][3] + "')";
songrun = true;
document.getElementById("queue").removeChild(elem);
70
sqc = sqc + 1;
function next_in_Q() {
songrun = false;
if (sqc == eqc) {
alert("Queue is empty.");
return;
}
var elem = document.getElementById("e" + sqc);
var xa = elem.name;
var pa = songs[xa][0];
document.getElementById("sname").innerHTML = sname[xa];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[x
a][3] + "')";
songrun = true;
document.getElementById("queue").removeChild(elem);
sqc = sqc + 1;
}
function rand_play() {
var index = Math.random() * path.length;
index = parseInt(index);
var pa = songs[index][0];
document.getElementById("sname").innerHTML = sname[index];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[i
ndex][3] + "')";
songrun = true;
function moody(val) {
var index = Math.random() * mood[val].length;
index = parseInt(index);
var pa = songs[mood[val][index] - 1][0];
document.getElementById("sname").innerHTML = sname[mood[val][index] - 1];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[m
ood[val][index] - 1][3] + "')";
71
songrun = true;
}
72
REFERENCE
73
(Lecture Notes in Computer Science), Berlin, Heidelberg, Springer: vol. 6422.
2010, pp. 122–131.
[7] Shivam Gupta, ‘‘Facial emotion recognition in real-time and static images,’’,
2018.
74