See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/367978045

Autism Spectrum Disorder Detection: Video Games based Facial Expression

Diagnosis using Deep Learning

Article in International Journal of Advanced Computer Science and Applications · January 2023
DOI: 10.14569/IJACSA.2023.0140112

CITATIONS READS

5 1,131

3 authors:

Morched Derbali Mu'tasem Jarrah

King Abdulaziz University UA
18 PUBLICATIONS 90 CITATIONS 10 PUBLICATIONS 146 CITATIONS

SEE PROFILE SEE PROFILE

Princy Randhawa
Manipal University Jaipur
41 PUBLICATIONS 246 CITATIONS

SEE PROFILE

All content following this page was uploaded by Morched Derbali on 02 May 2023.

The user has requested enhancement of the downloaded file.

 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

Autism Spectrum Disorder Detection: Video Games

based Facial Expression Diagnosis using
Deep Learning
Morched Derbali1, Mutasem Jarrah2, and Princy Randhawa3
Department of IT, Faculty of Computing and IT, King Abdulaziz University, Jeddah, Saudi1, 2
Department of Mechatronics Engineering, Manipal University Jaipur, Jaipur, India3

Abstract—In this study, a novel method is proposed for persistent differences in communication, interpersonal
determining whether a child between the ages of 3 and 10 has relationships, and social engagement [4]. Example: Being
autism spectrum disorder. Video games have the ability to nonverbal or having abnormal speech patterns, having trouble
immerse a child in an intense and immersive environment. With understanding nonverbal communication, developing and
the expansion of the gaming industry over the past decade, the maintaining relationships, and having trouble maintaining a
availability and customization of games for children has traditional back-and-forth conversational manner [5].
increased dramatically. When children play video games, they Repetitive habits, interests, and behaviors. Excessive
may display a variety of facial expressions and emotions. These awareness to or significantly reduced sensitivity to many
facial expressions can aid in the diagnosis of autism. Footage of
sensory stimuli, repetitive sounds or phrases (echolalia),
children playing a game may yield a wealth of information
regarding behavioral patterns, especially autistic behavior. You
preference for homogeneity and complexity with transition or
can submit any video of a child playing a game to the interface, regimen, rigid or heavily restricted and strenuous interests,
which is powered by the algorithm presented in this work. We hyper sensitivity to or dramatically reduced sensitivity to
utilized a dataset of 2,536 facial images of autistic and typically many sensory stimuli, rigid or severely regulated and intense
developing children for this purpose. The accuracy and loss interests, hyper sensitive. According to the American
function are presented to examine the 92.3% accurate prediction Psychological Association's Diagnostic statistical, autistic
outcomes generated by the CNN model and deep learning. traits must be present in early childhood, but they may not
fully express until social pressure builds the person's strength
Keywords—Autism in children; machine learning; deep to deal with them, and difficulties may be covered up by
learning; convolution neural network (CNN); video games; learned coping skills [6].
prediction
B. The Role of Video Games in Autism
I. INTRODUCTION According to research, enabling youngsters to play games
Autism is a complicated, behaviorally defined, static using smart phones might help detect autism. Dr. Jonathan
condition of an immature brain which is of significant concern Delafield-Butt, a senior professor in childhood development,
to practicing pediatricians due to a staggering 55.6 per-cent said it was important to detect autism early so parents and
rise in pediatric incidence from 1991 and 1997, surpassing children could receive a variety of support services. [7].
spina bifida, cancer, and Down syndrome [1]. Rather than new Autism is a neurodevelopmental disorder with many shared
environmental effects, this increase is due to increased traits, challenges, and abilities. Many autistic people have
awareness and evolving diagnostic criteria. Autism is a visual-spatial thinking, pattern identification, and a visual
condition with numerous nongenetic and genetic origins, preference. Games that require visual clues and spatial skills
rather than a disease. Autism (autism spectrum disorders) is are rewarding to such people. Games are creative but
defined as a group of developmental disorders characterized structured. RPGs and scrolling shooters satisfy research
by deficiencies in three behavioral domains: [2] participants' desire for imagination without requiring self-
generated creativity, which many autistic people lack. Video
1) interpersonal interaction. games have many audio and visual cues. Autistic people value
2) a diverse set of areas of interest and hobbies; and rules and objectivity more than neurotypical people. To avoid
3) speech, communication, and creative play. anxiety and sensory meltdowns, follow clear guidelines.
Video games reinforce clear expectations. Autistics need
A. Autism and its Characteristics: routine and repetition. Unknown circumstances cause anxiety,
Early childhood autism is a pervasive developmental discomfort, and a desire to escape. Video games allow for
disorder. Autism affects communication, relationships, and controlled practice and mastery [8,9]. Games are more
self-control. Infants often get autism. Autism is a "spectrum controlled than real life. Autism makes unpredictable human
disorder" that affects people differently. It has many traits. [3]. behavior difficult. Understanding social signs, idioms, humor,
Early diagnosis can help a person with autism live a full sarcasm, and satire can also cause anxiety. Playing a game that
life. According to the DSM-5, autism is characterized by becomes more familiar each time helps autistic gamers
overcome these challenges in a safe, controlled environment.

110 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

Parents and educators worry that autistic students spend too brain condition found in children and adolescents. Both autism
much time gaming instead of socializing. Playing has many spectrum disorder and attention deficit hyperactivity disorder
benefits if encouraged and controlled [10]. (ADHD) tend to continue into adulthood. A diagnosis of the
condition is made in approximately 5.9–9.4% of all infants.
II. RELATED WORK Because autism spectrum disorder (ASD) and attention deficit
Modern diagnostic tools for mental diseases were hyperactivity disorder (ADHD) are so common in children,
developed in the late 1800s, although their origins may be accurate and timely diagnosis of these conditions is critical.
dated back to the 4th century B.C (Before Christ) Era [11]. [15,16,17].
The gold standard for diagnosing often these mental-disorders The field of artificial intelligence known as machine
relies heavily on information gathered from various learning has the potential to significantly improve the use of
respondents (e.g., parents, teachers) about the onset, direction, computer methods in the field of neuroscience. A significant
and duration of various behavioral descriptors, which is then amount of research has been done to establish machine-
considered by providers when making a diagnosis predicated learning models and deep-learning approaches in order to
on DSM-5 (Diagnostic and Statistical Manual) Categorization interpret high-dimensional MRI (Magnetic Resonance
of Diseases-10th Installment (ICD-10) requirements [12]. Imaging) data in order to simulate neural networks that
Providers employ a variety of strategies to collect this data, regulate the brains of people who suffer from a variety of
ranging from subjective (e.g., assessment scale) and mental illnesses. [18,19]. These studies resulted in the
unstructured (e.g., semi - structured or unstructured development of machine-learning methods for the
interviews) to much more objective (e.g., actual observations) classification of Alzheimer's disease, mild cognitive
and organized (e.g., structured diagnostic interviews) [13]. impairment, right temporal epilepsy, schizophrenia,
Parkinson's disease, dementia, attention deficit hyperactivity
disorder, autism spectrum disorder, and major depressive
disorder [20]. These statistical algorithm-based machine-
learning models are well-suited to complex issues that require
a combinatorial explosion of options or non-linear processes.
This is because typical computer models fail in terms of
quality or scalability in these situations, but these statistical
algorithm-based machine-learning models succeed [21].
Senju et al, discussed the approaches to early detection of
autism in infants. Early detection here refers to before 18
months of age. It gives an overview of the known processes of
Fig. 1. Machine learning model for prediction and classification early social development in children that can fall under the
category in which “core deficits” are manifested in young
Fig. 1 shows the architecture of the machine learning children and summarizes a criterion for the same. The paper
model that is used commonly for prediction of machine concludes by discussing how the preferences for social stimuli
learning along with video games. The video of the child changes in the infants over time [22]. They develop a
playing the game is captured and then the data is preference for familiar people in first couple of months of life.
preprocessed, and machine learning is applied on it to predict They begin to have one to one interaction with their caregivers
the results. between three and six months of age. This is the period where
they understand the interactive styles of their usual social
Autism spectrum disorder (also referred to as ASD) and
partners. During four to seven months, infants can
attention deficit hyperactivity disorder (also referred to as
differentiate between emotional expressions of the face. Soon
ADHD) are both conditions that are relatively common in
they develop preferences for caregivers and after 12 months,
children and can continue into adulthood. Autism spectrum
they display a pattern of response to separation and reunion
disorder (ASD) is a developmental condition that causes
that demonstrates what they have learned to expect from their
patients to have difficulties with speech, behavior, and social
experiences with their caregivers [23].
interaction. Patients with ASD also tend to engage in
repetitive behavior, have impatience issues, and attention So, in the case of the infants with autism, one would
problems. Since the publication of the fifth edition of the speculate that the dyadic interactions would be less intense
Diagnostic and Statistical Manual of Mental Disorders (DSM- and lower capacity to recognize the emotional expressions of
5), the term autism spectrum disorder (ASD) has been used to others. However, by three to five years of age, autistic
refer to a more comprehensive diagnostic entity that formerly children would be capable of forming secure attachments to
referred to a number of distinct disorders, including Autistic their caregivers. The author concluded that similar patterns of
Disorder, Asperger's, and other Pervasive Chromosomal attachment security and insecurity are found in typically
Anomalies [14]. According to findings from recent studies, the developing children, it is very unlikely to be considered as a
prevalence of autism spectrum disorder (ASD) in children and core deficit in autistic infants. In infants, failure to
adolescents has increased from 1 in 100 to 1 in 59 in the past discriminate emotions can be considered as an important
14 years (from the year 2000 to 2014). Inattention, feature to characterize autism. But however, the rate of false
hyperactivity, and impulsivity are all symptoms of attention positives could be very high. Children with autistic tendencies
deficit hyperactivity disorder (ADHD), which is a common make use of different methods and skills to develop secure

111 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

attachments and to obtain the capacity to differentiate between Wu et al [28] analyzed the performances of various ML
the facial expressions and emotions than the typically techniques such as Bagging, Boosting, rule induction, and
developing children. decision tree classifiers on child, adolescent and adult ASD
screening datasets. The error rates of adult dataset for the
Whalen et al. [23,24] discovered that using a computer above techniques were between 5.68 and 8.23 per cent
game to teach children with autism improved motivation and whereas the Rules Machine Learning (RML) model
engagement when compared to the traditional methods of outperformed them with an error rate less than 5.6 per cent.
trying to teach children with autism. They established the Hence the paper was concluded by clearly revealing how ML
computer game Teach Town to assist autistic children improve approaches like covering can be used for obtaining promising
social, emotional, academic, and adaptive skills [25].Whalen's results [29].
findings are in line with what others in the area have found:
video-game-like tools are beneficial with students with ASD Jacob et al [29] obtained a high standard clinical data of
because they are constant and predictable, entail few social children at risk for ASD to implement machine learning
variables, and enable children to control and set the speed of algorithms. The aim is to build a low-cost and easy to use
the activity. Many children on the autism spectrum have ASD screening tool. To implement this, the author chose to
visual-spatial abilities that create videos actual gameplay an proceed with a combination of two approaches. Two different
area of expertise. While it's normal to be concerned about algorithms are trained to combine their outputs as a final
inattention, behavioral concerns, and even addictions, there screening assessment. One is based on the short, structured
really are simple ways that may help autistic children get the parent- reported questionnaires and the second is based on
most out of computer games and computers without causing tagging key behaviors from casual home videos of the test
additional problems at home or at school. The digital world subjects. The first classifier was trained using data from ADI-
provides a wealth of resources for teaching issue, social skills, R (Autism Diagnostic Interview) score sheets with labels
adaptability in new contexts, and even motor skill corresponding to established clinical diagnoses. The training
development. Raising autistic children may be difficult, but of second classifier i.e., the video classifier was done using
apps, games, and technologies can make contacting and ADOS (Autism Diagnostic Observation Schedule) instrument
educating autistic children simpler. Tbatah et al [26] laid out score sheets and diagnostic labels. To ensure sufficient
a principle called anger superiority hypothesis as its training volume, progressive sampling was used in both the
foundation. This hypothesis states that “angry faces capture cases. After evaluating multiple machine learning algorithms,
attention faster than happy faces” in common individuals. The the author chose Random Forests for its robustness against
authors aim to test and compare the threat detection abilities in overfitting [30].
autistic people with individuals with a history of typical
development using a facial visual search paradigm. In the clinical sample, the results showed that the parent
questionnaire classification approach performed better than
Thabtah et al [26] theory is based on building some of the more established screening tools, such as the M-
classification systems using machine learning, specifically a CHAT (Modified Checklist for Autism in Toddlers) and the
new method called Rules-machine Learning. This approach CBCL (Child Behavior Checklist). By combining the two
helps in detecting autistic traits and offers user knowledge different methods of classification into a single evaluation,
bases (rules) which enables the professionals to make better performance was improved. The author concluded by stating
analysis of the reasons behind the classification. how ML can play a crucial role in enhancing the performance
of the behavioral health screeners and how this research
The primary objective of this technique is rule discovery demonstrated a significant improvement over established
by search method which can be done using covering screening tools for autism. The author also mentioned how the
classification. Then, evaluation is performed on the discovered research demonstrated how this research demonstrated
rules to discard any redundancies and to optimize it further by significant improvement over established screening tools for
reducing the number of discovered rules. In order to improve autism. Chorianopoulou et al. [31] presented an ML-based
the overall efficiency of the training process, this phase approach to early diagnosis of ASD from videos of infants by
contributes to the narrowing of the search space for individual identifying specific behaviours from them. This approach was
data items. The classifier, which is utilized to make based on using videos of the infants. They used a dataset that
predictions regarding the value of the class, is more contained 2000 short videos with various behaviours of
comparable to an outcome of the rule evaluation phase interest, such as directed gaze towards faces or objects of
described above. For the purpose of generating the necessary interest, positive affect vocalization, and other similar
data from the participants, a mobile application known as behaviours, all of which were manually coded by expert raters
ASD Tests is utilized [27]. It implements four screening [32]. This dataset was used to conduct their research.
methods for toddlers, children, adolescents, and adults based
on the Q-CHAT-10 (Quantitative Checklist for Autism in The authors addressed the issue by employing a deep
Toddlers), AQ-10-child, AQ-10-Adolescent, and AQ-10 adult, learning model that was image-based and that was based on
respectively. The Q-CHAT-10 is a quantitative checklist for facial behavior features. Gorriz et al [32] has applied the
autism in toddlers. In addition to this, the author made use of various feature transformation techniques such as Log, Z-
the datasets that were previously deposited in the University score, sine functions to the collected datasets of toddlers,
of California Irvine Data Repository by the authors. children, adolescents, and adults [33]. In the next stage,
various classification techniques were implemented with these

112 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

transformed ASD datasets, to evaluate and assess their feature transformations. The mean highest accuracy of 97.2%
performance [33]. was achieved by Adaboost for Log and Scale, respectively.
Finally, the maximum highest accuracy of 100% was achieved
For toddler dataset, the median highest result was by all classifiers and feature transformation methods [34,35].
calculated by Adaboost for Log transformation, Adaboost and
SVM (Support Vector Machine) for Scale transformation For adolescent dataset, the median highest accuracy of
respectively as 99.06%. The mean highest result which was 95% was obtained by C5.0, LDA, PCA where LDA and PCA
reported to be 98.77% was calculated by SVM for Log and for both Log and Scale as well as C5.0 for Scale. The mean
Sine transformations. The maximum highest result was highest result of 93.89% was obtained by PCA for Log and
recorded to be 100% was calculated by Adaboost, GLMboost GLMboost for Scale, respectively. The maximum highest
and SVM for all feature transformation methods and C5.0 for result of 100% were achieved by all classifiers and feature
Scale transformation [27, 34]. transformation methods [35,36].
For child dataset, the median highest accuracy of 100% Table I summarizes previous studies and its findings and
was achieved by LDA (Linear Discriminant analysis) and limitations on usage of algorithms and the different features
PCA (Principal Component Analysis) for Log and Scale for the autism spectrum disorder
TABLE I. SUMMARIZED PREVIOUS STUDIES AND ITS FINDINGS FOR AUTISM SPECTRUM DISORDER

Reference Findings Limitations

Datasets were not balanced. The sample is
Thabtah et al, 2018 Inside the ASD screening tool, SVM was used to integrate the ML algorithm.
small comprising 612 autism patients and 11
[6] 97.6 % accuracy
non-autism instances.
Rules-Machine Learning is a machine learning approach based on rule induction
(RML). In terms of class labels,
Thabtah et al, 2020 Covering learning was used to generate non-redundant rules in a simple method. RML appeared to be ineffective when dealing
[26] RML classifies with greater prediction accuracy than typical algorithms such as with unbalanced data sets. There were no
boosting, bagging, and decision trees, thanks to the use of ten times cross- examples of toddlers in this article.
validation to split the dataset into ten subsets.
Optimal feature selection was automated using the Binary Firefly algorithm (ten In the ASD kid dataset, there were some
out of twenty-one features were chosen as the best). occurrences that are missing. There was a risk
Vaishali et al, 2018 There was no concern with class imbalance (there are 151 occurrences with class of model overfitting on the dataset because to
[27] 'yes' and 141 instances with class 'no' in the ASD youngster‟s dataset). Models the smaller number of occurrences in the
such as NB, J48, SVM, and KNN were used. SVM obtained the highest accuracy dataset. Swarm intelligence wrappers had
of 97.95 percent. certain drawbacks (Binary Firefly algorithm)
Analyzed the patient's condition using facial expressions and emotions,
employing an AI system and sensor data.
Al banna et al, 2020 Sent out frequent messages to parents, assisting the patient in coping with ASD When compared to other methods, the
during COVID-19. A smart wristband with an integrated monitor and camera is accuracy is poor. The research is still in its
[33] linked to a smartphone app in this system. Used real-time grayscale photos from early phases.
one Kaggle dataset of 35,887 images to detect ASD. The Inception-ResNetV2
architecture had the greatest accuracy of all the models, at 78.56 percent.
In contrast to earlier studies, the ML models
The authors devised a new algorithm that combined structural and functional
utilized demonstrate a 4.2 percent
Sen et al, 2018 characteristics. Drew many different depictions of the brain's functional
improvement in the accuracy of the
[37] connections. The results showed that incorporating multimodal characteristics
predictions for Autism. Datasets suffer
improves case discrimination accuracy the most.
significantly from fluctuations.
SVM, Naive Bayes, and Random Forest classification algorithms are used.
There were 95,577 kid records with 367 variables, of which 256 were deemed to Doesn't predict the severity of ASD. A
Van den et al, 2017 be adequate. Different qualities were well delineated. Created a dataset with four cursory collection of traits (criteria) used to
[39] classifications (ASD: None, Mild, Moderate, Severe). The J48 algorithm attained identify ASD, which may or may not always
the highest accuracy of 87.1 percent (2 class) and 54.1 percent (4 class) (decision correspond to an instance of ASD.
tree)
The methodology followed can be divided into five steps
III. METHODOLOGY as shown in Fig. 2:
This study demonstrated the use of Deep Learning and
1) Capturing the facial expressions while playing a Video
Image processing techniques for the detection of Autism using
facial expressions. The initial approach was to build and train Game.
a neural network based on the available data on Autism. 2) Data Preprocessing.
Following this, any video of the patient which clearly shows 3) Model Building and Training.
their facial expressions could be taken as the input through an 4) Prediction and Optimization.
interface created for the users. This input was used for the 5) Uploading the video to the Web Interface.
detection of autistic characteristics using the previously
trained model [38].

113 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

2) Fix target size: The collected image dataset had to be

resized for uniformity in processing and to do so, all the
images were resized to 64x64 pixel so that the convolution
neural network model can be trained using those images [42].
This was done by using the function “flow_from_directory”
and the argument to the function changed to 64x64 pixels.
Whereas the neural network was training on the training data,
the flow from directory () function was used to read photos
straight from the directory and enhance them. The technique
assumed that photos from various classes stored in separate
directories but are all included within the same parent
directory.
3) Horizontal flip: The images needed to be flipped
horizontally to maintain uniformity, so this is achieved by
using the function “ImageDataGenerator” and adjusting the
arguments.
4) Rescaling: The ImageDataGenerator class may rescale
pixel values from 0-255 to the recommended 0-1 range for
neural network models. Normalization is the process of
reducing data to a number between 0 and 1. Setting the rescale
parameter to a ratio that may be multiplied by each pixel to
produce the required range will do this [41,42].
C. Model Building and Training:
Deep learning is a technique for automatic learning that
Fig. 2. Work flow of the method for the detection of facial expression while
playing video games
implements the use of examples to teach machines how to
learn in the same way that people do. A self-driving car's
A. Capturing the Facial Expressions while Playing the Video ability to recognize a stop sign or differentiate between a
Game: pedestrian and a lamppost is dependent on its use of deep
learning, which is an essential component of the technology. It
The facial expression of the child is captured while she/he makes it possible to control consumer electronics with one's
is playing the video game using the web camera attached to voice, such as mobile phones and tablets, televisions, and
the personal computer. The video is captured and saved hands-free speakers. The concept of "deep learning" has been
locally and then the same video is uploaded to the website receiving a lot of attention as of late, and with good reason.
user interface and then the trained Convolutional Neural It's about achieving things that weren't possible before you
Network (CNN) model predicts if the child in the image started working on them.
broken down from the video is autistic or not [40].
During the process of deep learning, a computer model
B. Data Preprocessing will learn to perform categorization tasks directly from either
The collected image dataset was preprocessed by 3 pictures, text, or sound. Models that use deep learning have
processes so that the CNN model can train the dataset to the potential to achieve an accuracy that is on par with or even
predict if the child is autistic or not. They were: exceeds that of humans in certain circumstances. In order to
train models, a significant amount of labelled data as well as
1) Dividing the video into frames: OpenCV was used to various topologies of multilayer neural networks are utilized.
fragment the video captured into images or frames. The video The acronym "CNN" stands for "Convolution Neural
is captured using a webcam while the child is playing the Network". Image recognition and processing are two
game to monitor and analyze the video captured which would applications that make use of a type of artificial neural
be broken down into images and then train the CNN model. network known as a convolutional neural network (also
The dataset consists of 2536 images belonging to the autistic abbreviated as CNN). CNNs are designed to focus specifically
and non-autistic classes in the training set and 300 images in on analyzing pixel input. CNN was used to train the model
the test set. The required image data is ob-tained from Kaggle. based on the image data sets that were taken by the webcam of
The ImageDataGenerator class of keras library enables us to the computer that the child was using to play the video game
on. CNNs are image processing, artificially intelligent (AI)
read the images from the folders. Rescaling of images is done
systems that utilize deep learning to perform both generating
by dividing each pixel value of an image by 255. The images and informative tasks. These tasks frequently include machine
in the dataset are of various sizes which calls for resizing into vision, which includes image and video identification, as well
one final size of 64x64 pixel and the same can be as recommendation systems for natural language processing
implemented using a function called “flow_from_directory” of (NLP) [40, -42]. A neural network is a piece of computer
the above class [40,41]. hardware and/or software that mimics the way neurons in the

114 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

human brain communicate with one another. Traditional this optimizer. Adam optimization is a gradient descent
neural networks were not intended to be used for image approach based on adaptive first- and second-order moment
analysis, so in order for them to do so, they require the images estimation. The approach is "computationally more efficient,
to be broken up into smaller chunks. The "neurons" that make has small memory demand, is robust to diagonally resizing of
up CNN are organized more similarly to those in the gradient, and is well suited for situations with huge
prefrontal cortex, which is the part of the brain in humans and data/parameters," as according to Thabtah- et al., 2020.
animals that is responsible for processing visual input. The
difficulty of processing images in pieces that is inherent to E. Uploading the video to the Web Interface
traditional neural networks can be circumvented by arranging Streamlit is one of the recent and fastest python-based
the layers of neurons in such a way that they cover the entirety model deployment tools. This open-sourced python based
of the visual field [48]. A CNN makes use of a technology framework simplified the whole model deployment cycle
similar to a perceptron that is designed to have minimal along with providing an easy way to structure the
requirements for processing [43]. The layers of a CNN are functionalities of the interface [45,46].
comprised of an input layer, a layer, and a hidden layer. These
layers are followed by several convolutional layers, average
pooling, fully connected layers, and normalizing layers. A
system that is significantly more effective and easier to train
for image analysis and natural language [43] has been made
possible as a result of the elimination of constraints and
improvements in the efficiency of image processing.
D. Prediction and Optimization
Optimization plays a crucial role for any machine learning
problem. Gradient descent is an optimization algorithm that
finds the lowest possible value or the minimum value of a
function through iterations. While the loss function which is
also known as the cost function is all about calculating the
loss/errors for every prediction that the neural network makes,
gradient descent can be used to find the minimum of this loss
function. The goal is to estimate the values of coefficients of a
function that can minimize the cost function. In other words,
the new coefficients will have a significantly lower cost.
This technique is initialized by taking small random values
as coefficients for the function. The cost is then evaluated by
inserting them into the function. The next step is to change the
values of the coefficients in a direction that can lead to a lower
cost in the next iteration. This direction can be estimated with Fig. 3. Web interface for prediction of the disease
the help of derivatives. The derivative gives a slope (gradient)
at the desired point on a curve, or a function and the sign of Once the neural network was trained, the model of an
that slope can be used to determine the direction in which the epoch with the best validation accuracy was saved as a
coefficients can be moved in further iterations [44]. Now that „.model‟ file. The interface enables the users to select the type
the algorithm is aware of the direction of progression of of media file that is to be uploaded. If the selected media type
coefficients with the help of the gradient at current position, is an image, the saved model is used to classify the uploaded
the next move is to make a step by scaling it and subtracting image to Autistic/Non-Autistic for which the result can be
the obtained value from the current position. Subtracting is displayed on the interface [45]. But if the selected media type
done as the aim is to minimize the function [40]. Another is a video, then the uploaded video is divided into frames
parameter called learning rate is used to scale the gradient and using OpenCV and each image is classified into Autistic/Non-
control the step size. Learning rate can affect the performance autistic with the help of the saved model. The mean value of
in a significant way. Smaller learning rate can lead the classification of all images is considered as the final
algorithm to reach the final iteration before even reaching the classification for the entire video and displayed on the
optimum point [41]. The Adam algorithm is implemented by interface.

115 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

Challenge [47 -48]. This model gets 92.7% top-5 accuracy on

ImageNet's 14 million photos from 1000 classes. 300 images
were used for the testing and 100 images were used for
validation of the CNN model [48].
The model achieved 92.3% accuracy for the testing dataset
and 87.3% accuracy for the validation dataset. For this study,
many evaluation metrics were calculated, and the results are
summarized in the following sections. The results of the
comparative prediction analysis of our model and existing
model of the same dataset as shown in Table II.

TABLE II. COMPARISON OF THE RESULTS USING THE SAME DATASET

Author Model Accuracy
Haque et al [49] ResNet50 89.2%

Shaik et al [50] VGG19 84.0%

Our model VGGFace 92.3%

A. Evaluation Metrics
Fig. 4. Prediction mechanism if autistic or not
Sensitivity, Specificity, Precision, Recall, F1 Score and
Fig. 3 describes the entire working of the prediction Accuracy were calculated for evaluating the performance of
mechanism implemented in this project. The video of the child the CNN model. The convention True Positive (TP), False
while playing the game is captured by a webcam and then the Positive (FP), True Negative (TN) and False Negative (FN)
video is saved locally. The same video is then uploaded to the were used.
website interface for the CNN deep learning algorithm to 1) Sensitivity: Sensitivity measures a model's ability to
work on it. After processing the file and after the prediction
recognize positive cases. TPR is sometimes called recall.
model is applied on the video, the outcome or result of the
prediction is shown on the interface as “Autistic” or “Non- Sensitivity measures how many positive cases a model
Autistic” as shown in Fig. 4. accurately identified. High-sensitivity models have minimal
false negatives, meaning they overlook some positive
IV. RESULTS AND DISCUSSION examples. Sensitivity is a model's capacity to recognize good
This study explores the use of video games to discriminate cases. We need our models to locate all good examples to
between children with and without ASD. Compared to generate accurate forecasts. True positive rate plus false
previous qualitative techniques, the activities and learning negative rate equals 1. Higher true positive rate means model
metrics in the evaluation game give a quantifiable depiction of correctly identifies positive situations. The sensitivity was
children's abilities, making the identification of ASD more calculated using the following formula as shown in eq. (1):
accurate and practical. The use of these games as a
Sensitivity = (1)
supplementary tool in educational interventions for kids with
ASD is also possible. In previous studies, the model has been The Sensitivity of the model was 0.9560 or 95.60%.
trained using different algorithms such as SVM, Neural
Network, RML Classifier, Random Forest, Naïve Bayes with 2) Specificity: Specificity assesses the model's ability to
a maximum accuracy of 97.5% [46]. They have used images identify real negatives. This means there will be a percentage
of a facial expression and behavior of a patient while in our of true negatives forecasted as positives, or false positives.
study we trained the model using CNN when the child is True Nega-tive Rate (TNR). True negative rate plus false
playing video game and capture the image of his behavior and positive rate equals 1 always. Low specificity suggests the
facial expression. The Convolutional Neural Network (CNN) model is mislabeling a lot of negative data as positive. The
model is used to extract and produce the proper patterns of the
specificity was calculated using the following formula as
face features when the child is playing video game and capture
the image of his behavior and facial expression such as shown in eq. (2):
inappropriate snickering and laughing, Lack of pain Specificity = (2)
sensitivity, Inability to maintain proper eye contact, unable to
communicate with gestures, Inadequate reaction to sound etc. The specificity of the model was 0.8865 or 88.65%.
The model was trained in the cloud using Google Colab with 3) Precision: Precision is the ratio of True Positives to
python, which supports TensorFlow and Keras. The epoch
total positive samples (either correctly or incorrectly). It is
number declared was 159 and it had a batch size of 20. The
VGG model was used for implementing the convolutional calculated using the formula as shown in eq. (3):
neural network. Karen Simonyan & Andrew Zisserman of Precision = (3)
Oxford University's Visual Geometry Group (VGG) proposed
VGG models, which performed well in the ImageNet The precision of the model was 0.9048 or 90.48%.

116 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

4) Accuracy: Model accuracy measures which model is a particular set of test data. As the model cannot afford to
better at finding correlations and patterns in a dataset based on predict non-autistic when the patient is autistic, the confusion
training data. Accuracy is calculated using the formula as matrix was calculated as shown in Fig. 5, which indicates that
shown in eq. (4): the false positive condition case value should be lower. 155 of
Accuracy = (4) the 300 images were correctly classified.

The accuracy was found out to be 92.3% for the testing

dataset consisting of 300 images and 87.3% for the validation
dataset.
5) F1 score: The harmonic mean of accuracy and recall is
used to get the F1 score. It is calculated using the formula as
shown in eq. (5):
F1 Score = 2 * (5)
The F1 Score was calculated to be 0.9297 or 92.97%.
6) Confusion matrix: A matrix called the confusion matrix
is used to assess how well classification models perform given
Fig. 5. Confusion matrix using CNN for autism spectrum disorder

Fig. 6. a) and b) . Model Training Accuracy and Training Loss

Fig. 6(a) and (b) depicts the plot of the model‟s training identify ASD. A precise autism diagnosis can help pick a
accuracy, training loss, respectively for 10 epochs. The treatment plan for autistic children. More precision might
highest accuracy achieved was 92.3 % with the VGG model improve autism diagnosis. The platform might reveal this
by applying two dense layers with specific parameters. A loss neurological disease, bringing treatment closer. This research
function optimizes ML algorithms. The loss is determined is part of how humans utilize technology to tackle the world's
during training and validation, and its interpretation depends healthcare issues. Future studies may use machine learning
on how well the model performs. It's the total of training or and deep learning algorithms to help individuals recognize a
validation set mistakes per example. Loss value indicates range of ailments using the same platform. Although young,
model's performance after each optimization cycle. A digital technologies offer unlimited potential. In the case of
performance metric measures the algorithm's accuracy. Model autism, a mix of digital tools and in-person therapy visits is
accuracy is generally estimated as a percentage based on expected. Autism therapy isn't one-size-fits-all. These tools
model parameters. It measures how well your model predicts will help researchers produce medicines for ASD patients
actual data. more swiftly. Language is being studied for potential
therapies. SFARI-funded researchers are employing
V. CONCLUSION cellphones and automatic transcription software to record
The study built a deep learning web app to diagnose speech from autistic youngsters. Based on the data collected
autism using a convolutional neural network and camera by recording footage of children playing video games, a video
footage of a youngster playing a video game. CNN's game can be designed for children with autism or autistic
architecture can extract facial attributes by generating facial symptoms. This game will help parents or guardians
feature patterns and assessing facial landmark distances, determine whether their kid has autism such as Kinect game.
classifying faces as autistic or not. VGG CNN Model
produced accurate results. Testing accuracy was 92.3%,
validation accuracy was 87.3%, and precision was 90.4%.
Future research will improve this model by broadening
psychologists' autistic kid diagnoses. This programme helps

117 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

ACKNOWLEDGMENT artificial intelligence,” in Proceedings of the International Conference on

Brain Informatics, pp. 251–262, Springer, Padua, Italy, July 2020.
The authors extend their appreciation to the Deputyship for [14] Ganesh, K.; Umapathy, S.; Thanaraj Krishnan, P. Deep learning
Research & Innovation, Ministry of Education in Saudi Arabia techniques for automated detection of autism spectrum disorder based on
for funding this research work through the project number thermal imaging. Proc Inst Mech Eng H 2021, 235, 1113-1127,
doi:10.1177/09544119211024778.
528752-G: 243-611-1443 and King Abdulaziz University,
DSR, Jeddah, Saudi Arabia. [15] Deshpande, G., Wang, P., Rangaprakash, D., and Wilamowski, B.
(2015). Fully connected cascade artificial neural network architecture for
Institutional Review Board Statement: Not applicable. attention deficit hyperactivity disorder classification from functional
magnetic resonance imaging data. IEEE Trans. Cybernet. 45, 2668–
Informed Consent Statement: Not applicable. 2679. doi: 10.1109/TCYB.2014.2379621
[16] A. V. Dahiya, C. McDonnell, E. DeLucia, and A. Scarpa, “A systematic
Data Availability Statement: Publicly available datasets review of remote telehealth assessments for early signs of autism
were analyzed in this study. This data can be found here: spectrum disorder: video and mobile applications,” Practice Innovations,
https://www.kaggle.com/code/basmarg/vgg-16-autism-image- vol. 5, no. 2, pp. 150–164, 2020.
data/data [17] Hassouneh, A.; Mutawa, A.M.; Murugappan, M. Development of a
Real-Time Emotion Recognition System Using Facial Expressions and
Conflicts of Interest: The authors declare no conflict of EEG based on machine learning and deep neural network methods.
interest. Informatics in Medicine Unlocked 2020, 20,
doi:10.1016/j.imu.2020.100372.
REFERENCES [18] Gwyn, T.; Roy, K.; Atay, M. Face Recognition Using Popular Deep Net
Architectures: A Brief Comparative Study. Future Internet ,2021, 13,
[1] Aljero, M.K.A.; Dimililer, N. A Novel Stacked Ensemble for Hate 164
Speech Recognition. Applied Sciences 2021, 11,
doi:10.3390/app112411684. [19] N. Zaman, J. Ferdus and A. Sattar, "Autism Spectrum Disorder
Detection Using Machine Learning Approach," 2021 12th International
[2] Suman Raj, Sarfaraz Masood, Analysis and Detection of Autism Conference on Computing Communication and Networking
Spectrum Disorder Using Machine Learning Techniques, Procedia Technologies (ICCCNT), Kharagpur, India, 2021, pp. 1-6, doi:
Computer Science, Volume 167, 2020, Pages 994-1004, ISSN 1877- 10.1109/ICCCNT51525.2021.9579522.
0509, https://doi.org/10.1016/j.procs.2020.03.399.
[20] Elshoky, B. R. G., Younis, E. M. G., Ali, A. A., and Ibrahim, O. A. S.,
[3] Chae, S.; Choi, A.; Jung, H.; Kim, T.H.; Kim, K.; Mun, J.H. Machine Comparing automated and non-automated machine learning for autism
Learning Model to Estimate Net Joint Moments during Lifting Task spectrum disorders classification using facial images, ETRI Journal 44
Using Wearable Sensors: A Preliminary Study for Design of (2022), 613– 623. https://doi.org/10.4218/etrij.2021-009721.
Exoskeleton Control System. Applied Sciences 2021, 11,
doi:10.3390/app112411735. [21] Sato, W.; Sawada, R.; Uono, S.; Yoshimura, S.; Kochiyama, T.; Kubota,
Y.; Sa-kihama, M.; Toichi, M. Impaired detection of happy facial
[4] Nishio, M. Special Issue on “Machine Learning/Deep Learning in expressions in autism. Sci Rep 2017, 7, 13340, doi:10.1038/s41598-017-
Medical Image Processing”. Applied Sciences 2021, 11, 11900-y.
doi:10.3390/app112311483.
[22] Lee JH, Lee GW, Bong G, Yoo HJ, Kim HK. Deep-Learning-Based
[5] Pană, M.-A.; Busnatu, Ș.-S.; Serbanoiu, L.-I.; Vasilescu, E.; Popescu, Detection of Infants with Autism Spectrum Disorder Using Auto-
N.; Andrei, C.; Sinescu, C.-J. Reducing the Heart Fail-ure Burden in Encoder Feature Representation. Sensors. 2020; 20(23):6762.
Romania by Predicting Congestive Heart Failure Using Artificial https://doi.org/10.3390/s20236762
Intelligence: Proof of Concept. Applied Sci-ences 2021, 11,
doi:10.3390/app112411728. [23] Levy, S.; Duda, M.; Haber, N.; Wall, D.P.J.M.a. Sparsifying machine
learning models identify stable subsets of predictive features for
[6] Thabtah, Fadi, Firuz Kamalov, and Khairan Rajab. (2018) "A new behavioral detection of autism. 2017, 8, 1-17.
computational intelligence approach to detect autistic features for autism
screening." International journal of medical informatics 117: 112-124. [24] Król, M.E.; Król, M.J.N. A novel machine learning analysis of eye-
tracking data reveals suboptimal visual information extraction from
[7] Tamang, L.D.; Kim, B.W. Deep Learning Approaches to Colorectal facial stimuli in individuals with autism. 2019, 129, 397-406.
Cancer Diagnosis: A Review. Applied Sciences 2021, 11,
doi:10.3390/app112210982. [25] Center for Autism Research. Social Responsiveness Scale, 2nd Edition
(SRS-2). Available online: https://www.carautismroadmap.org/social-
[8] Abbas, H.; Garberson, F.; Glover, E.; Wall, D.P. Machine learning responsiveness-scale/?print=pdf (accessed on 26 October 2020).
approach for early detection of autism by combining questionnaire and
home video screening. Journal of the American Medical Informatics [26] Thabtah, F.; Peebles, D. A new machine learning model based on
Association : JAMIA 2018, 25, 1000-1007, doi:10.1093/jamia/ocy039. induction of rules for autism detection. Health Informatics J 2020, 26,
264-286, doi:10.1177/1460458218824711.
[9] Akter, T.; Shahriare Satu, M.; Khan, M.I.; Ali, M.H.; Uddin, S.; Lio, P.;
Quinn, J.M.W.; Moni, M.A. Machine Learning-Based Models for Early [27] Vaishali, R., & Sasikala, R. (2018). A machine learning based approach
Stage Detection of Autism Spectrum Disorders. IEEE Access 2019, 7, to classify autism with optimum behaviour sets. International Journal of
166509-166527, doi:10.1109/access.2019.2952609. Engineering & Technology, 7(4), 18.
[10] M. N. Parikh, H. Li, and L. He, “Enhancing diagnosis of autism with [28] Wu, C.; Liaqat, S.; Helvaci, H.; Cheung, S.S.; Chuah, C.N.; Ozonoff, S.;
optimized machine learning models and personal characteristic data,” Young, G. Machine Learning Based Autism Spec-trum Disorder
Frontiers in Computational Neuroscience, 2019. Detection from Videos. Healthcom 2021, 2020,
doi:10.1109/healthcom49281.2021.9398924.
[11] Chu, H.-C.; Tsai, W.W.-J.; Liao, M.-J.; Chen, Y.-M. Facial emotion
recognition with transition detection for students with high-functioning [29] Jacob, S.; Wolff, J.J.; Steinbach, M.S.; Doyle, C.B.; Kumar, V.; Elison,
autism in adaptive e-learning. Soft Computing 2017, 22, 2973-2999, J.T.J.T.p. Neurodevelopmental heterogeneity and computational
doi:10.1007/s00500-017-2549-z. approaches for understanding autism. 2019, 9, 1-12.
[12] F. Thabtah and D. Peebles, “A new machine learning model based on [30] Chorianopoulou, A.; Tzinis, E.; Iosif, E.; Papoulidi, A.; Papailiou, C.;
induction of rules for autism detection,” Health Informatics Journal, vol. Potamianos, A. Engagement detection for children with Autism
26, no. 1, pp. 264–286, 2020. Spectrum Disorder. In Proceedings of the 2017 IEEE International
Conference on Acoustics, Speech and Sig-nal Processing (ICASSP), 5-9
[13] M. H. Al Banna, T. Ghosh, K. A. Taher, M. S. Kaiser, and M. Mahmud, March 2017, 2017; pp. 5055-5059.
“A monitoring system for patients of autism spectrum disorder using

118 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

[31] Achenie, L.E.; Scarpa, A.; Factor, R.S.; Wang, T.; Robins, D.L.; [42] Sumi, A.I.; Zohora, M.F.; Mahjabeen, M.; Faria, T.J.; Mahmud, M.;
McCrickard, D.S.J.J.o.d.; JDBP, b.p. A machine learning strategy for Kaiser, M.S. fassert: A fuzzy assistive system for chil-dren with autism
autism screening in toddlers. 2019, 40, 369. using internet of things. In Proceedings of the International Conference
[32] Górriz, J.M.; Ramírez, J.; Segovia, F.; Martínez, F.J.; Lai, M.-C.; on Brain Informatics, 2018; pp. 403-41
Lombardo, M.V.; Baron-Cohen, S.; Consortium, M.A.; Suckling, [43] Hassan, T.-u.; Abbassi, R.; Jerbi, H.; Mehmood, K.; Tahir, M.F.;
J.J.I.j.o.n.s. A machine learning approach to reveal the neurophenotypes Cheema, K.M.; Elavarasan, R.M.; Ali, F.; Khan, I.A. A Novel
of autisms. 2019, 29, 1850058. Algorithm for MPPT of an Isolated PV System Using Push Pull
[33] Al Banna, M.H.; Ghosh, T.; Taher, K.A.; Kaiser, M.S.; Mahmud, M. A Converter with Fuzzy Logic Controller. Energies 2020, 13,
monitoring system for patients of autism spectrum disorder using doi:10.3390/en13154007.
artificial intelligence. In Proceedings of the International Conference on [44] Tariq, Q.; Fleming, S.L.; Schwartz, J.N.; Dunlap, K.; Corbin, C.;
Brain Informatics, 2020; pp. 251-262. Washington, P.; Kalantarian, H.; Khan, N.Z.; Darmstadt, G.L.; Wall,
[34] Duda, M.; Ma, R.; Haber, N.; Wall, D.P.J.T.P. Use of machine learning D.P.J.J.o.m.I.r. Detecting developmental delay and autism through
for behavioral distinction of autism and ADHD. 2016, 6. machine learning models using home vide-os of Bangladeshi children:
Development and validation study. 2019, 21, e13822.
[35] Sen, B.; Borle, N.C.; Greiner, R.; Brown, M.R.J.P.o. A general
prediction model for the detection of ADHD and Autism using structural [45] Pavithra, D., Jayanthi, A.N., Nidhya, R. and Balamurugan, S. (2022).
and functional MRI. 2018, 13, e0194856. Autism Screening Tools With Machine Learning and Deep Learning
Methods: A Review. In Tele-Healthcare (eds R. Nidhya, M. Kumar and
[36] Thabtah, F.; Peebles, D.J.H.i.j. A new machine learning model based on S. Balamurugan). https://doi.org/10.1002/9781119841937.
induction of rules for autism detection. 2020, 26, 264-286.
[46] P. Mazumdar, G. Arru, and F. Battisti,Early detection ofchildren with
[37] Sen, B., Borle, N. C., Greiner, R., & Brown, M. R. (2018). A general autism spectrum disorder based on visual explora-tionof images, Signal
prediction model for the detection of ADHD and Au-tism using Process. Image Commun.94(2021), 116184.
structural and functional MRI. PloS one, 13(4), e0194856..
[47] Fawaz Waselallah Alsaade, Mohammed Saeed Alzahrani,
[38] F. C. Tamilarasi and J. Shanmugam, “Convolutional Neural Network
"Classification and Detection of Autism Spectrum Disorder Based on
based Autism Classification,” in 2020 5th Interna-tional Conference on
Deep Learning Algorithms", Computational Intelligence and
Communication and Electronics Systems (ICCES), pp. 1208–1212,
Neuroscience, vol. 2022, Article ID 8709145, 10 pages, 2022.
IEEE, (2020, June).
https://doi.org/10.1155/2022/8709145.
[39] van den Bekerom, B. Using machine learning for detection of autism
[48] Miao, Y., Dong, H., Jaam, J. M. A., & Saddik, A. E. (2019). A deep
spectrum disorder. In Proceedings of the Proc. 20th Student Conf. IT,
learning system for recognizing facial expression in real-time. ACM
2017; pp. 1-7.
Transactions on Multimedia Computing, Communications, and
[40] Carette, R.; Elbattah, M.; Cilia, F.; Dequen, G.; Guérin, J.-L.; Bosche, J. Applications (TOMM), 15(2), 1-20.
Learning to Predict Autism Spectrum Disorder based on the Visual [49] M. I. U. Haque and D. Valles, “A facial expression recognition approach
Patterns of Eye-tracking Scanpaths. In Proceedings of the HEALTHINF, using DCNN for autistic children to identify emotions,” in Proceedings
2019; pp. 103-112.
of the 2018 IEEE 9th Annual Information Technology, Electronics and
[41] Guimarães, A.J.; Araujo, V.J.S.; Araujo, V.S.; Batista, L.O.; de Campos Mobile Communication Conference (IEMCON), pp. 546–551,
Souza, P.V. A hybrid model based on fuzzy rules to act on the diagnosed Vancouver, Canada, 1–3 November 2018.
of autism in adults. In Proceedings of the IFIP International Conference
[50] Jahanara, S., & Padmanabhan, S. (2021). Detecting autism from facial
on Artificial Intelligence Applications and Innovations, 2019; pp. 401-
image, International Journal of Advance Research, Ideas and
412.
Innovations in Technology, Volume 7, Issue 2 - V7I2-1181.

119 | P a g e
www.ijacsa.thesai.org
View publication stats