UMT Artificial Intelligence Review (UMT-AIR)

Volume 3 Issue 1, Spring 2023

ISSN(P): 2791-1276 ISSN(E): 2791-1268
Homepage: https://journals.umt.edu.pk/index.php/UMT-AIR

Article QR

Multi-Modal Data Fusion for Classification of Autism Spectrum

Title:
Disorder Using Phenotypic and Neuroimaging Data
Author (s): Adnan Younas, Muhammad Yousuf Kamal, Sumaira Kausar, Samabia Tehsin

Affiliation (s): Center of Excellence in Artificial Intelligence (COE-AI), Department of Computer

Science, Bahria University, Islamabad, Pakistan
DOI: https://doi.org/10.32350.umt-air.31.01

History: Received: November 3, 2022, Revised: December 8, 2022, Accepted: March 28, 2023,
Published: June 2, 2023
Citation: A. Younas, M. Y. Kamal, S. Kausar, and S. Tehsin, “Multi-Modal data fusion for
classification of autism spectrum disorder using phenotypic and neuroimaging
data,” UMT Artif. Intell. Rev., vol. 3, no. 1, pp. 01–16, June 2023, doi:
https://doi.org/10.32350.umt-air.31.01

Copyright: © The Authors

Licensing: This article is open access and is distributed under the terms of
Creative Commons Attribution 4.0 International License
Conflict of
Interest: Author(s) declared no conflict of interest

A publication of
Department of Information System, Dr. Hasan Murad School of Management
University of Management and Technology, Lahore, Pakistan
Multi-Modal Data Fusion for Classification of Autism Spectrum
DisorderUsing Phenotypic and Neuroimaging Data
Adnan Younas, Muhammad Yousuf Kamal, Sumaira Kausar *, and Samabia
Tehsin
Center of Excellence in Artificial Intelligence (COE-AI), Department of
Computer Science, Bahria University, Islamabad, Pakistan
ABSTRACT Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that
causes disrupted social behaviors and interactions of individuals. Hence, it can adversely
affect the social functioning of individuals. Eachautistic individual is said to have a sort of
unique behavioral pattern. ASD has three major sub-categories, namely autism, Asperger,
and pervasive developmental disorder, not otherwise specified. The term spectrum
indicates that ASD possesses a large variety of symptoms of severity. Practitioners need to
have a vast experience and expertise for the accurate analysis of the symptoms of ASD.
These symptoms need to be acquired from a range of modalities. An accurate diagnosis
requires the analysis of brain scan and phenotypic data. These aspects present a multifold
challenge for computer-aided ASD diagnosis. Most of the existing computer aided ASD
diagnosis systems are capable of diagnosing only whether an individual is affected with
ASD or not. A detailed categorization into the subcategories of ASD in such diagnosis is
missing. Another aspect that is missing in the existing techniques is that symptoms are
observed from a single modality. This can adversely affect the accuracy of diagnosis, since
different modalities focus on different aspects of symptoms. These challenges and gaps
provided the motivation to present a method that covers the variety exhibited in ASD, while
considering the dire need of acquiring symptoms from a variety of data sources. The
proposed method showed rather encouraging results. Moreover, the achieved results are
evident of the efficacy of the proposed method.
INDEX TERMS Asperger, Autism Spectrum Disorder (ASD), diagnosis, feature
fusion, machine learning, psychiatry
JEL CODES H51, H52, and H53
I. INTRODUCTION that affects the verbal and social skills of an
individual with no discrimination of age,
A person with autism lacks social
gender, race, or any other social
communication and is involved in
background. Autism growth around the
repetitive behaviors [1]. The term
globe has made researchers work on early
‘spectrum’ describes the range which may
diagnosis and treatment to make the
vary from mild to severe levels of
effected individual an active part of society
disabilities in skills and behaviors [2].
again. Automated diagnosis of ASD is of
Regardless of race, ethnicity, culture, and
dire need of the time. Early and speedy
socioeconomic background, the basic signs
diagnosis can help patients in their early
of Autism Spectrum Disorder (ASD) are lack
treatment. With the emergence of AI
of social interaction and repetitive behavior
technology, the treatment of ASD has
[2]. ASD is a neurodevelopmental disorder

*
Corresponding Author: [email protected]
UMT Artificial Intelligence Review
2
Volume 3 Issue 1, Spring 2023
 Younas et al.

become possible that would not only be inter- view revised), and “DISCO” (a
beneficial for patients, however, for all diagnostic instrument for social
other stakeholders as well. communication disorders, UK) are
available[2]. Some other tools are,
According to the diagnostic and statistical
“modified checklist for autism in toddlers
manual of mental disorder 4th edition
revised, with a follow up” (M-CHAT-R/F)”
(DSM-IV), autism has different types
survey of the well-being of young
including autistic disorder, Asperger
children”. Some other tools are also
disorder, childhood disintegrative disorder,
available to measure social deficiencies,
and pervasive developmental disorder not
such as “Social Communication
otherwise specified [3, 4]. This was the
Questionnaire (SCQ)”, “Social
actual diagnostic classificationof DSM-IV
Responsiveness Scale (SRS)” and “Autism
published by the American Psychiatric
Spectrum Screening” questionnaire. All
Association in 1994. In 2013, DSM-IV was
such tools are based on personal
upgraded to 5th edition as DSM-5 in which
observations, interviews, and
all these types except Rett syndrome were
questionnaires which are fairly
combined to make it ”spectrum” with a
questionable in the context of reliability.
range from mild to severe [1]. Different
Several societies prefer genetic testing in
studies have discovered that both, genetic
which different laboratory tests are
and environmental factors are responsible
performed, such as CBC, (complete blood
for restriction in the development of the
count), urine examination, and stool
brain and may cause autism due to
analysis. Brain images are not a common
changes \ in cerebellar architecture and
practice, however, autism’s relationship
abnormalities in the limbic system [1]. No
with brain urges th e clinicians to get
clear causes are found for autism, however,
neuroimages through MRI for detailed
there are several misconceptions about its
analysis [1].
causes [5]. There are some common
parental beliefs regarding the causes of With the advancement in the field of
autism. Parents believe that ASD in their neuroscience and psychopathology, early
child is either due to their child’s brain detection of ASD is possible [7]. ASD has
structure, environmental pollution, and some similar symptoms to other disorders,
genetic problems, or maybe the will of God. such as ADHD that makes screening
Some parents associate it with generalized difficult. Early detection of autism can help
stress, bad luck, poor diet, and tobacco or clinicians treat the patient at an early stage.
alcohol consumption [6]. There is no [8] ASD is a result of genetic mutation [9].
connection between vaccination and autism Most studies suggest that very few patients
[5], however, many parents feel that are diagnosed at an early age, although it is
vaccines have toxins that cause autism. proven in studies that diagnosis under the
Diagnosis of autism is not that easy since age of three has a stability rate of 100%.
clinicians have to depend on personal [10] Several studies have used automated
observation and information provided by methods based on computer vision
parents. [5, 6]. techniques and data analysis for diagnosis
of ASD [7–9].
Several screening tools including “STAT”
(screening tool for autism in toddlers), ASD is diagnosed clinically with different
‘’ADOS” (autism diagnostic observation tools including STAT, ADOS, ADI-R,
schedule), “ADI-R” (autism diagnostic CARS, SRS, and SCQ [2]. Iidka [12]
Department of Information Systems
3
Volume 3 Issue 1, Spring 2023
Multi-Modal Data Fusion for Classification…

applied neural networks to classify teenager 64.23% was achieved. The study focused
ASD patients and obtained 90% accuracy. on binary classification. [20] applied a deep
Chen et al. [13] used SVM and classified neural network for the diagnosis of ASD
79.17% of cases accurately. [14] obtained patients by using brain images dataset. Their
73.4% accurate results with CNN in which study employed a hybrid model of
a novel approach was proposed by using unsupervised autoencoders and supervised
full resolution 3D spatial structure of rs- CNN and obtained an accuracy of 84.05%.
MRI data. Moreover, ABIDE dataset was Pominova et al. [21] performed domain
also used for binary classification and an adaptation on brain images data for the
accuracy of 73.3% was obtained. Li et al. classification of ASD patients based on
[15] applied deep neural networks in a 2-stage brain pathology. Their approach
method for the classification of ASD in outperformed other existing approaches
which fMRI images were used through a with the use of 3D convolutional
3D CNN sigmoid classifier with an autoencoders. Lu et al. [22] proposed a
accuracy of 85.3%.Moreover, the problem fuzzy multi-kernel clustering approach
of interpretation of reliable biomarkers was based on autoencoders and an accuracy of
also addressed which was related to ASD 61% was obtained by combining thefMRI
classification, however, rs-FMRI images and phenotypic data. Their clustering
were used for binary classification. approach performed better than others for
the diagnosis of ASD.
Heinsfeld et al. [16] identified the most
influential areas of brain that causes ASD Huang et al. [23] used ABIDE-I multi-site
with an accuracy of 70% by using DNN. and multi-template data and classified ASD
Autoencoder increases the performance of patients by using brain image features and
model and classifiers, such as RF, SVM, achieved an accuracy of 89.13% .In 2018,
and DNN showed accuracies of 63%, 65%, Khosla et al. [9] used functional MRI and
and 70%, respectively. Brain images from obtained an accuracy of 73.3% for binary
ABIDE dataset were used to differentiate classification of ASD patients. However, a
ASD patients from non-autistic people. large number of studies have been conducted
Yang et al. [17] classified ASD and TD for boosting the performance of autism
with the help of rs-fMRI by using tensor classification by using different data
flow-based DNN models and obtained an processing techniques [24].
accuracy of 75.27%. Resting- state fMRI
It is evident from the literature of
was used, acquired from multi-site, through
automated ASD diagnosis that some areas
ABIDE repository for conducting the
require more focus in the research. One of
study. Moreover, images data was used for
the key observation in this regard is that
binary classification of ASD and no ASD.
majority of the work in literature is focused
Behavioral- based features were not used.
on binary classification, that is, ASD
Yin et al. [18] reviewed fMRI and sMRI
effected and control classes. Very less to
based diagnosis of ASD.
none work has been done in the
Arya et al. [19] used feature fusion of classification of sub categories of ASD.
behavioral and brain images data from and Another related research dimension that
with the help of GCN framework. They requires more attention is multi-modal data
fused brain summaries obtained from 3D analysis for diagnosis. The current study
CNN with phenotypic data to make the focused on these two aspects. The proposed
model more effective. A mean accuracy of method focuses on classification into
UMT Artificial Intelligence Review
4
Volume 3 Issue 1, Spring 2023
 Younas et al.

subcategories of ASD and it also Some other sites have images data sets but
considered multiple modalities to take they lack behavioral and functional data.
symptoms. Moreover, the achieved results This data set comprises of clinical as well
were promising. behavioral data sets. In many other data
sets, data is available just for infants,
II. METHODS
adolescents, or adults separately, however,
A. DATASET here in this initiative, all ranges of age are
covered in a single data set. Moreover, most
Many datasets are available for ASD datasets are related to men, while ABIDE
patients. Kaggle, UCI machine learning, deals with both genders. Hence, global
and different self-collected datasets are
participation has made it more diverseand
available on different sites. However, an
effective in generalizing the diagnosis
initiative by the “National Institute of
process around the globe.
Mental Health America” has made a
repository of autism patients’ data with This data repository is presented by
controls. This data is publicly available on International Neuroimaging Data-sharing
the NIMH website for further research.It is a Initiative(INDI). It has two data sets named
multi-modal data with considerable number of ABIDE-I and ABIDE-II. Each data set
records. It is a multi-site global ASD data comprises of brain images taken at different
collected from different states of America laboratories over theworld. Hence, it should
and other sites over the world. This data be taken into consideration that these
features brain images that plays an images are taken with different MRI
important role in the diagnosis of ASD . machines and in different settings.ABIDE-
I was released in 2012 with 1112 records of

17.
FIGURE 1. Proposed model
Department of Information Systems
5
Volume 3 Issue 1, Spring 2023
Multi-Modal Data Fusion for Classification…

Figure 1 proposed model global sites. It B. DATA PRE-PROCESSING

includes resting state functional magnetic
Data pre-processing is a necessary step to
resonance imaging (rs-fMRI), phenotypic
go through before feeding the data into
data with functional data from INDI. It
main classifier. Unprocessed data creates
comprises of 573 individuals with no
many ambiguities in the proper
autism (controls) and 539 ASD patients
classification. Processed data works more
with an age ranging from 7-64 for both
effectively and efficiently. In the data
genders. In June 2016, ABIDE initiative
description, several missing values were
released its second edition with 521 ASD
mentioned along with some follow-ups and
patients with 593 controls with an age
sessional samples that were included in
ranging from5-64 from 19 global sites. This
different modalities. DTI modality was
data set had 1114 total records with more
excluded due to less number of records,
phenotypic features, especially correlated
while others still had many imbalances. For
with ASD and its symptoms. ABIDE-II
this purpose, data pre-processing was
data set presented morepsychopathological
conducted. A single session was used and
phenotypic features to dis- criminate ASD
all follow-ups were skipped. All values
from other similar diseases. [25] ABIDE-II
with missing records in any modality were
data set had 1114 records collectively. In
removed. Moreover, any feature with
individual anatomy modal (Anatqap) had
missing records was not considered to
1359 records including follow-up and
avoid any misinterpretation. Each modality
records of longitudinal samples.
was individually treated and records were
Functional modal had 1449 records
removed that were missing in other
including follow-up sessions as well ,while
modalities. Figure 1 explains the pre-
composite modal comprising of clinical
processing steps taken in this study.
observations and behavioral data related to
any individual who contributed to data 1) FEATURE SELECTION
collection research had 1114 records. To
make it universal, extra records of follow- First of all, all the missing values were
up and longitudinal samples were checked and removed. Due to medical data,
eliminated. it was not possible to replicate the data
augmentation since it can be misguiding
Behavioral and clinical diagnosis is and may misinterpret the results. Each
included in a composite file with a large modality was dealt separately. In this
number of personal, behavioral, and DSM- modality, threedifferent scores regarding IQ
5 manual records. ABIDE-II has were included. However, due to missing
psychopathological data of patients and values its average was decided to be used.
controls with brain images numerical AIQ score was made and all average scores
features. There are two modalities for this were put in this column. After all the pre-
purpose. One is Anatqap which is an processing, 1026 records were obtained in
anatomy modality It contains structural all three files which were the same and
brain images features, while the other is complete in all modalities. Autism was
functional qap which contains functional classified into sub-classes of autistic,
features of the brain of ASD patients and Asperger, and PDDs-NOS as discussed by
controls. DSM-IV which was later combined to make
autism a ”spectrum” with mild to severe
range. All the missing values were removed

UMT Artificial Intelligence Review

6
Volume 3 Issue 1, Spring 2023
 Younas et al.

and 589 records were obtained with sub- D. CLASSIFICATION

class classification labels. The current
Classification refers to make groups of data
study focused to classify ASD patients into
according to particular labels.
further sub-classes of autistic, Asperger,
Classification is performed for separating
and PDDs-NOS. For the three-class multi-
objects differently from others. Machine
classification of ASD patients without
learning can be used for automated
controls, another data subset of ASD patients
classification of ASD subcategories. In
with 271 records was obtained.
literature, majority of the work is focused
2) DATA NORMALIZATION on binary classification of patients into
autistic and control classes. In the current
After dealing with missing values and
study, three different experimental setups
feature selection, data normalization was
were employed for classification.
performed since there was a diverse variety
of data ranges having different features. 1.Binary classification was performed to
Some attributes had very small values classify ASD patients andcontrols.
ranging between 0 and 1, while others had
2. Multiclass classification was performed
range of hundreds. This diversity can
for four categories of subjects, that is, no
adversely affect the classification method.
autism, autistic, Asperger, and PDDs-NOS.
Therefore, data normalization was
performed. There are several techniques to 3. Sub-classification of ASD patients was
perform data normalization. Min Max performed, that is, ASD patients into their
approach was employed to normalize the sub-categories as mentioned in DSM-IV.
data between 0 and 1. The proposed classification setup can be
summarized as:
C. FEATURE LEVEL FUSION OF
DATASETS 𝐶𝐶(𝑋𝑋) → 𝑦𝑦 where C is the classifier
The process of data fusion is X= Xs U Xf U Xp
commonly performed to get the combined
effect of attributes. Sometimes, data → ∈ → and ��∈ → , and ��∈ →
𝑋𝑋𝑠𝑠 𝑺𝑺 𝑋𝑋𝑓𝑓 𝑭𝑭 𝑋𝑋𝑝𝑝 𝑷𝑷
inferences are incomplete without some
𝒔𝒔𝒔𝒔
other attributes. Hence, fusion helps to fill →= : (Structural MRI indicators)
this gap. Fusion of data is performed at a 𝑺𝑺
𝒔𝒔𝒔𝒔
different level. It can be performed at the
feature level as well as at the final stage of 𝒇𝒇𝒇𝒇
decision, as [1] proposed a model of →= : (Functional MRI indicators)
𝑭𝑭
adaptive boundaries by using fusion of 𝒇𝒇𝒇𝒇
classifiers which improved the results 𝒑𝒑𝒑𝒑
significantly [1]. Feature level fusion is the →= : (Phenotypic indicators)
combination of features acquired from 𝑷𝑷
𝒑𝒑𝒑𝒑
different modalities. This fusion helps to get
the collective effect of these diverse y∈ →
𝒀𝒀
attributes and hence, makes the
𝒚𝒚𝒚𝒚
classification more robust, effective, and → = (for Binary classification)
accurate. In this way, features are observed 𝒀𝒀 𝒚𝒚𝒚𝒚
collaboratively and the decision is made
based on all combined attributes.
Department of Information Systems
7
Volume 3 Issue 1, Spring 2023
Multi-Modal Data Fusion for Classification…

𝒚𝒚𝒚𝒚 distributed among 412 records for training,

→ = 𝒚𝒚𝒚𝒚 (for 3 class classification) 59 records for validation, and 118 records
𝒀𝒀
𝒚𝒚𝒚𝒚 for testing purposes. In performing the sub-
classification of ASD patients without
controls, there were 271 records which,
𝒚𝒚𝒚𝒚 again, was split with the same ratio as 189
→ = : (for4 class classification) samples for training, 28 samples for
𝒀𝒀
𝒚𝒚𝒚𝒚 validation, and 54 records for testing the
model. Each ML model would be discussed
Table I describes the number of
with different hyper-parameters. The
samples for binary, 4-class multiclass
settings for each classifier would be
classification, and 3-class multiclass
discussed in-dependently.
classification.
C. CLASSIFICATION
III. EXPERIMENTS
Machine learning methods are used
A. EXPERIMENTAL SETUP
for ASD classification. Experimentation
ABIDE-II dataset was used for the was performed on three well established
experimentation. “Google Collaborator” classifiers, that is, Decision tree (DT),
(Google Colab) was used to perform the Support Vector Machine (SVM), and
experiments. The virtual machine which Artificial Neural Networks (ANN).
was used for Collaboration had 13GB
TABLE I
RAM and 2 v CPU with 2-core Xeon
2.2GHz. To evaluate the performance of the CLASSIFICATION WITH TOTAL
models, a classification report was used to NUMBER OF RECORDS
mention accuracy with precision . The
weighted average score of all measures of No. Classification Samples Classes
precision were compared andrecalled. 1 Binary Class 1026 2-Class
B. DATASET 2 Multi-Class 589 4-Class
ABIDE dataset showed total samples ASD Sub-
of 1114 with 521 ASD patient samples and 3 271 3-Class
Class
593 control records. After pre-processing,
1026 records were obtained. For the 1) DECISION TREE (DT)
training of classification models, the data
was split into 70% for training, 10% for A Decision Tree (DT) is a machine
validation, and the remaining 20% was learning model that is used as a classifier to
used for testing the model. In some classify the binary or multiclass method. It
experiments, a 60-10-30 training- examines the goodness of given attributes
validation- test split was also used, to perform classification. Best node takes
however, results with a 70- 10-20 ratio the root position and rest are ranked and
were encouraging. For binary positioned in the tree according to their
classification, the data was distributed goodness value. Root nodes are the classes.
including 718 samples for training, 103 Purity of the node is the criteria for the
samples for validation, and 205 samples for goodness of the nodes. To conduct the
testing. In multi-class classification, there current study, purity measures were used,
were 589 records and the data was
UMT Artificial Intelligence Review
8
Volume 3 Issue 1, Spring 2023
 Younas et al.

that is, entropy and Gini index. The formula Linear: K(x,y) = x.y
for both is given below:
Polynomial: K(x,y) = (1+x.y)d
𝑛𝑛
Gini= 1 − �𝑖𝑖=1 𝑝𝑝2 (ci) RBF: K(x,y) = exp(-a||x-y||)2
𝑛𝑛
Entropy = �𝑖𝑖=1 −𝑝𝑝 (ci) * log2 (p (ci))
2
Sigmoid: K(x,y) = tanh(ax.y+b)
Where, p (ci) is probability or percentage of 3) NEURAL NETWORKS (NN)
class (ci) in a node. In DT, “Gini” and A neural network (NN) is one of the
“Entropy” were used as criteria with a max most popular classifier. Ittakes inspiration
depth of 5. The model was fine-tuned with from the structure and working of human
different parameters and resultantly, the brain. The idea behind neural network was
model performed well with 70% training “neuron” which is a basic unit of brain.
samples and ”Gini” as a criterion with a There is some activation function that
depth of 5. processes the data with checking of error
2) SUPPORT VECTOR MACHINES and updating its weight. Gradually,
(SVM) machine learns and error is minimized. In
NN, an activation function refers to a
Support Vector Machine (SVM) is a mathematical functionthat maps inputs to the
machine learning model used for neuron to output of that neuron. Table II
classification. It works with mathematical shows different activation functions with
functions called ”Kernel”. This method equations and derivatives.
makes vectors of different groups along a Backpropagation is used to learn the
threshold and each valueis grouped into its parameters of the network.
particular vector. SVM is one of the leading Backpropagation uses gradients to optimize
classifiers. In SVM, different kernels are the parameters.
used. Four different kernels were used
including Linear, Polynomial,Sigmoid, and In the NN model, a simple model
Radial Base Function (RBF). Different having input, hidden, and output was used.
hyper-parameters were also used. The For binary classification, sigmoid in output
value of “C” error term w a s a l s o s e t and rectified linear unit (Relu) function in
in linear to50 and in RBF to 100 i n hidden layers were used. The Tangent
o r d e r to avoid over-fitting.In RBF, the Hyperbolic function in input and hidden
value of “gamma” was set to 1 in order to layers was also tested. To compute the loss,
make sure that the model uses a low value Binary Cross entropy with “ Adam” as
to get the influence of each training an optimizer was used. Stochastic gradient
example as far as possible. In polynomial descent (SGD) was also tested, however, it
kernel, the polynomial degree was set to 8 didn’t perform well. The batch size w a s
as going to higher values takes enough time s e t to 200 with 1000 epochsto in order
to run one instance. The train test split was to run the model for sufficient training and
70-10-20, however, 60-10-30 ratio was also prediction. 10-fold cross-validation was
experimented. usedto enhance model efficiency. In multi-
class classification, the same model was
Some of the basic kernels used in SVM are used with the same parameters, however,
given below with their mathematical the activation function was changed to
equations. “Softmax” for multi-class classification.

Department of Information Systems

9
Volume 3 Issue 1, Spring 2023
Multi-Modal Data Fusion for Classification…

TABLE II
DERIVATIVES OF ACTIVATION FUNCTIONS
No Function Equation Derivative
1
1 Sigmoid σ(𝑥𝑥) = 𝑓𝑓′(𝑥𝑥) = 𝑓𝑓(𝑥𝑥)(1 − 𝑓𝑓(𝑥𝑥))
1 − 𝑒𝑒 −𝑥𝑥
𝑒𝑒 𝑥𝑥 − 𝑒𝑒 −𝑥𝑥 𝑓𝑓′(𝑥𝑥) = 1 − 𝑓𝑓(𝑥𝑥)2
2 TanH σ(𝑥𝑥) =
𝑒𝑒 𝑥𝑥 + 𝑒𝑒 −𝑥𝑥
0 𝑖𝑖𝑖𝑖 𝑥𝑥 < 0 0 𝑖𝑖𝑖𝑖 𝑥𝑥 < 0
𝑓𝑓(𝑥𝑥) = � 𝑓𝑓′(𝑥𝑥) = �
3 ReLu 𝑥𝑥 𝑖𝑖𝑖𝑖 𝑥𝑥 ≥ 0 1 𝑖𝑖𝑖𝑖 𝑥𝑥 ≥ 0

𝑒𝑒 𝑥𝑥 𝑒𝑒 𝑥𝑥 (𝑒𝑒 𝑥𝑥 )2
4 Softmax 𝑓𝑓(𝑥𝑥) = 𝑓𝑓 ′ (𝑥𝑥) = − 𝑗𝑗
∑𝑗𝑗1 +𝑒𝑒 𝑥𝑥 ∑𝑗𝑗1 +𝑒𝑒 𝑥𝑥 (�1 +𝑒𝑒 𝑥𝑥 )2

IV. RESULTS A. ANALYSIS OF FEATURE FUSION

Performance evaluation of machine All three models were evaluated through
learning models is mandatory to assess the evaluation metrics. The dataset was
model efficiency. The current study, imbalanced, therefore reliability on
worked on accuracy, precision, and recall. accuracy was not sufficient and it was
Accuracy is the most common measure better to check other measures, such as
used to assess the performance of classifiers precision and recall. In feature fused
in machine learning. However, when the modality, weighted averages of all these
data set is imbalanced, accuracy cannot measures were obtained. The results in
predict model performance alone and it is binary classification showed that SVM and
important to consider other measuresto get neural networks performed better than a
a clear picture of model performance. The DT. About 95±3 % were obtained in each
current study used accuracy, precision, and measure of accuracy, precision, and recall
recall to get a deep insight into model in SVM. In the NN, the best results of
performance since the data was 97±1% were obtained, however, 81±2% in
imbalanced, however, with a little DT for all measures. In 4-Class Multi-
skewness. classification, DT performance was better
than both SVM and NN. DT showed 83±3%
T P +T N
and SVM showed83% in precision and 82%
Accuracy =
accuracy and recall. However, NN showed a
T P + F P + T N + FN
77% result. It has been observed that DT’s
T P precision and accuracy were better than
Precision = SVM and NN. Finally, in 3-Class Multi-
T P +F P classification of ASD patients without
controls, NN, DT, and SVM performed
T P
nearly similarly and showed 89% in both
Recall = SVM and NN, while 87% accuracy and
T P +F N recall in DT with88% precision. Here, DT
results remained a little lower than SVM
and NN for all measures.
UMT Artificial Intelligence Review
10
Volume 3 Issue 1, Spring 2023
 Younas et al.

The first step of the current study was to approach. Results showed thatthe approach
classify ASD and controls which remained outperformed without feature fusion in DT
the focus of previous studies. Targeting Dx and SVM. Behavioral data showed the
group with two labels, samples were same results in NN in precision, however,
classified in individual modal and in the remained lower in accuracy and recall.
proposed method of feature fusion modal. Results showed that feature fusion gave
The current study proved that the proposed better results in all classifiers.
approach outperformed individual
Finally, the main focus of the current study
modalities in all classifiers. Results showed
was to classify ASD patients into
that feature fusion of multi-modality data
subcategories which were mostly not
gave the best results. The next step was to
discussed in the previous studies. “PDD
classify ASD and controls into broad multi-
DSM IV” were targeted with three labels.
classes of no autism, autistic, Asperger,and
Samples were classified in individual
PDDs-NOS, which remained a gap in
modalities and in the proposed method of
previous studies. Targeting “PDD DSM
feature fusion approach. Achieved results
IV” withfour labels, samples were classified in
showed that feature fusion gave the better
individual modalities as well as in the
results.
proposed method of feature fusion

FIGURE 2. Analysis of feature fusion

B. ANALYSIS OF FEATURE outperformed all individual modalities.
SELECTION With an average of 95±3% in all measures,
fused dataset performed best both in SVM
The current study attempted to use feature
and NN. However, in DT, its performance
engineering for fused data in order to
remained at 82±3%, however, still, it is
improve the results, however, the research
better than individual modality results.
showed that the use of PCA and auto
SVM, DT, and NN performed differently in
encoder was not helpful in this case.
binary classification. In the proposed
Experiments proved that feature reduction
method, NN remained at the top with an
declined in NN to 55%. Finally, the
average of 98% in each measure, while
proposed approach of feature-fused modal
Department of Information Systems
11
Volume 3 Issue 1, Spring 2023
Multi-Modal Data Fusion for Classification…

SVM showed 92% overall. However, DT DT remained best at 69±2%. The

results declined to 63±4% in all individual behavioral data has an average performance
modalities. It declined the results further, of 54 ±2% in NN for each measure of
proving that no more feature engineering accuracy and recall. T h e behavioral data
was helpful and preprocessing was done modal performed better in SVM than NN
already which was good enough to classify which showed 56% result in all measures.
the data. Any reduction or transformation Behavioral data results declined adversely
of the features would not be helpful and itis in DT to 44% accuracy and recall, while
better to use features in the same space precision showed 49%. Finally, the
since any such effort can adversely affect proposed approach was used and the results
theoverall classification. of feature fused modal performed best than
all individualmodalities with an average of
V. DISCUSSION
89±1% for all measures in SVM and NN.
ABIDE-II has different modalities of The fused dataset performed best both, in
anatomy, functional and behavioral data. SVM and NN, However, in DT its
The experiments conducted on all performance showed a slight difference of
modalities individually provethat fusion of 1% with 88% for precision and 87%
all features performs better thanconsidering accuracy and recall. Here, still, it is better
just functional features of brain images data than individual modalities in precision and
(anatomy). accuracy. The main objective of this study
was to classify ASD patients into further
The purpose of conducting this subclasses. In the proposed method, DT
research was to sub-classify the ASD
showed an average of 87±1% accuracyand
patients into sub-categories. At final level,
recall in feature fused modality, however,
control samples w e r e r e m o v e d with
the resultsof DT declined badly in anatomy
thes e l e c t i o n o f ASD patients to and behavioral modalities. In behavioral
classify them into autistic, Asperger, and and functional modality, accuracy and recall
PDDs-NOS. The experiments conducted on
remained at 51±2%. While, overall SVM
all modalities individually proved that
performed best with 62±2% in individual
fusion of all features performs better than
modalities for each measure except for the
considering just behavioral or brain images
fused feature approachwhere SVM results
which remained the focus of the majority outperformed all individual modalities with
of previous studies. In ASD patient sub- 89% for each measure. While, NN results
classification, the anatomy modality
were similar for each measure in all
showed 62% accuracy and recall in SVM
functional and behavioral modalities
with 64% precision. However, its results
,leveled at 56% except fused modality
declined in DT to 51%. While, precision in
where results remained best with 89% and
NN was enhanced to 63%. Functional outperformed individual modalities. Figures
modality performed best in DT with 71% 3, 4, and 5 show accuracy, precision, and
precision and 69% in all other measures, recall for all modalities with each classifier
however, the results in NN declined to performance.
56%. Functional modality performance in

UMT Artificial Intelligence Review

12
Volume 3 Issue 1, Spring 2023
 Younas et al.

FIGURE 3. Inter modality accuracy comparison in binary and multi-class classification

FIGURE 4. Inter modality precision comparison in binary and multi-class classification

FIGURE 5. Inter modality recall comparison in binary and multi-class classification

Arya et al. [19] classified ASD patients by conducted by Huang et al. [23] used
using feature fusion of fMRI and sMRI ABIDE-I multi-site, multi-template data,
features of ABIDE-I+II data with a mean and classified ASD patients by using brain
accuracy of 64.23%. The proposed feature image features showing an accuracy of
fusion method classified ASD with a 98% 89:13% and recall of 91%. In 2018, Khosla
accuracy. A study conducted by Rakic´ et et al. [14] used functional MRI from both
al. [24] also used ABIDE-I by fusing fMRI ABIDE-I and II and obtained an accuracy
and sMRI features at decision level and of 73.3%for binary classification of ASD
diagnosed ASD patients with an ensemble patients. Figure 6 represents a comparative
accuracy of 85.06%. The research result summaryon ABIDE data fusion.

Department of Information Systems

13
Volume 3 Issue 1, Spring 2023
Multi-Modal Data Fusion for Classification…

FIGURE 6. Comparative result summary on ABIDE data fusion

A. CONCLUSION varying range of” mild” to ”severe”. Hence,
the use of machinelearning models to multi-
The current study achieved its objective to
modality data, in order to estimate the
providing an effective automated diagnosis
severity range of the autism spectrum
of ASD and its sub-classification. The
would be a challenging task and can be a
proposed method of feature fusion of multi-
prospect for researchers.
modalities data can help in more accurate
and effective sub-classification of ASD REFERENCES
patients. Results showed that this feature
[1] H. Hodges, C. Fealko, and N. Soares,
level fusion is more effective in the
“Autism spectrum disorder:
diagnosis and sub-classification of ASD
Definition, epidemiology, causes, and
patients. This method can help clinicians in
clinical evaluation,” Transla. Pediat.,
cross-validation of their clinical diagnosis
vol. 9, no. Suppl 1, pp. S55–S65, Feb.
with the results of speedy ML models. The
2020, doi: https://doi.org/10.
diagnosisof ASD with 98% results is good
21037/tp.2019.09.09
enough to be used even in clinical practices,
while classification into subcategories with [2] A. Mannion and G. Leader,
89% result is alsoquite encouraging. There “Attention-deficit/hyperactivity
are some future directions to work withthe disorder (AD/HD) in autism spectrum
ABIDE-II dataset. Although, the data set disorder,” Res. Autism Spec. Disord.,
provides enough information, however, vol. 8, no. 4, pp. 432–439, Apr. 2014,
there are stillways for improvement. Due to doi: https://doi.org/10.1016/j.rasd.
its heterogeneity, multi-modality data can 2013.12.021
be used for generalizing ASD diagnosis
[3] N. Yirmiya and T. Charman, “The
including developingcountries’ data which
prodrome of autism: early behavioral
is not included in this repository yet.
and biological signs, regression, peri-
Additionally, many attributes in behavioral
and post-natal development and
modality have missing records and if these
genetics,” J Child Psychol. Psych., vol.
missing records are made available, it may
51, no. 4, pp. 432–458, Jan. 2010, doi:
help to take more attributes from behavioral
https://doi.org/10.1111/j.1469-7610
data which hasa key role in ASD diagnosis.
.2010.02214.x
Moreover, in DSM-5, ASD is considered as
a spectrum andpatients are classified with a
UMT Artificial Intelligence Review
14
Volume 3 Issue 1, Spring 2023
 Younas et al.

[4] Md. M. Rahman, O. L. Usman, R. C. 146045821882471, Jan. 2019, doi:

Muniyandi, S. Sahran, S. Mohamed, https://doi.org/10.1177/146045821882
and R. A. Razak, “A Review of 4711
machine learning methods of feature
[10] W. Guthrie, L. B. Swineford, C.
selection and classification for autism
Nottke, and A. M. Wetherby, “Early
spectrum disorder,” Brain Sci., vol. 10,
diagnosis of autism spectrum disorder:
no. 12, Art. no. 949, Dec. 2020, doi:
stability and change in clinical
https://doi.org/10.3390/brainsci10120
diagnosis and symptom
949
presentation,” J. Child Psychol.
[5] R. P. Goin-Kochel et al., “Beliefs Psych., vol. 54, no. 5, pp. 582–590,
about causes of autism and vaccine Oct. 2012, doi: https://doi.org/10.
hesitancy among parents of children 1111/jcpp.12008
with autism spectrum
[11] S. Woolfenden, V. Sarkozy, G. Ridley,
disorder,” Vaccine, vol. 38, no. 40, pp.
and K. Williams, “A systematic review
6327–6333, Sep. 2020, doi:
of the diagnostic stability of Autism
https://doi.org/10.1016/j.vaccine.2020
Spectrum Disorder,” Res. Autism
.07.034
Spect. Disord., vol. 6, no. 1, pp. 345–
[6] C. M. Brewton et al., “Parental beliefs 354, Jan. 2012, doi: https://doi.org
about causes of autism spectrum /10.1016/j.rasd.2011.06.008
disorder: An investigation of a
[12] T. Iidaka, “Resting state functional
research measure using principal
magnetic resonance imaging and
component analysis,” Res Autism
neural network classified autism and
Spect. Disord., vol. 87, Art, no.
control,” Cortex, vol. 63, pp. 55–67,
101825, Sep. 2021, doi: https://doi.org/
Feb. 2015, doi: https://doi.org/10.
10.1016/j.rasd.2021.101825
1016/j.cortex.2014.08.011
[7] A. Z. Guo, “Automated autism
[13] C. P. Chen et al., “Diagnostic
detection based on characterizing
classification of intrinsic functional
observable patterns from
connectivity highlights
photos,” IEEE Trans. Affect. Comput.,
somatosensory, default mode, and
2020, doi: https://doi.org/10.1109
visual regions in
/taffc.2020.3035088
autism,” NeuroImage, vol. 8, pp. 238–
[8] J. N. Constantino and T. Charman, 245, 2015, doi: https://doi.org
“Diagnosis of autism spectrum /10.1016/j.nicl.2015.04.002
disorder: reconciling the syndrome, its
[14] K. Khosla, K. Jamison, A. Kuceyeski,
diverse origins, and variation in
and M. R. Sabuncu, “3D convolutional
expression,” Lancet Neurol., vol. 15,
neural networks for classification of
no. 3, pp. 279–291, Mar. 2016, doi:
functional connectomes,” in Proc.
https://doi.org/10.1016/s1474-4422
Deep Learn. Med. Image Anal.
(15)00151-9
Multimodal Learn. Clinic. Deci.
[9] F. Thabtah and D. Peebles, “A new Granada, Spain, Sep. 20, 2018, pp.
machine learning model based on 137–145.
induction of rules for autism
[15] X. Li, N. C. Dvornek, J. Zhuang, P.
detection,” Health Inform. J., Art. no.
Ventola, and J. S. Duncan, “Brain
Department of Information Systems
15
Volume 3 Issue 1, Spring 2023
Multi-Modal Data Fusion for Classification…

biomarker interpretation in ASD using Conf. Mach. Vision, Rome, Italy, Nov.
deep learning and fMRI,” In Int. Conf. 2–6, 2021, pp. 570–577, doi:
Med. Image Comput. Comput-assist. https://doi.org/10.1117/12.2587348
Interven., pp. 206-214. Granada,
H. Lu, S. Liu, H. Wei, and J. Tu,
Spain, Sept. 16-20, 2018, pp. 206–214.
[22]

“Multi-kernel fuzzy clustering based

[16] A. S. Heinsfeld, A. R. Frano, R. C. on auto-encoder for fMRI functional
Craddock, A. Buchweitz, and F. network,” Expert Syst. Appl., vol. 159,
Meneguzzia, “Identification of autism Art. no. 113513, Nov. 2020, doi:
spectrum disorder using deep learning https://doi.org/10.1016/j.eswa.2020.11
and the ABIDE dataset,” NeuroImage, 3513
vol. 17, pp. 16–23, Jan. 2018, doi: [23] F. Huang et al., “Self-weighted
https://doi.org/10.1016/j.nicl.2017.08. adaptive structure learning for ASD
017 diagnosis via multi-template multi-
center representation,” Med. Image
[17] X. Yang, P. T. Schrader, and N. Zhang,
Anal., vol. 63, Art. no. 101662, Jul.
“A deep neural network study of the
2020, doi: https://doi.org/10.
abide repository on autism spectrum
1016/j.media.2020.101662
classification,” Int. J. Adv. Comput.
Sci. Appl., vol. 11, no. 4, 2020, doi: [24] M. Rakić, M. Cabezas, K. Kushibar, A.
https://doi.org/10.14569/ijacsa.2020.0 Oliver, and X. Lladó, “Improving the
110401 detection of autism spectrum disorder
by combining structural and functional
[18] W. Yin, S. Mostafa, and F. Wu,
MRI information,” NeuroImage, vol.
“Diagnosis of autism spectrum
25, Art. no. 102181, 2020, doi:
disorder based on functional brain
https://doi.org/10.1016/j.nicl.2020.102
networks with deep learning,” J.
181
Comput. Biol., Oct. 2020, doi:
https://doi.org/10.1089/cmb.2020.025 [25] A. Di Martino et al., “Enhancing
2 studies of the connectome in autism
using the autism brain imaging data
[19] D. Arya et al., “Fusing structural and
exchange II,” Sci. Data, vol. 4, no. 1,
functional MRIs using graph
Art. no. 170010, Mar. 2017, doi:
convolutional networks for autism
https://doi.org/10.1038/sdata.2017.10
classification,” in Med. Imag. Deep
Lear., 2020, pp. 44–61.
[20] H. Sewani and R. Kashef, “An
Autoencoder-Based deep learning
classifier for efficient diagnosis of
autism,” Children, vol. 7, no. 10, Art.
no. 182, Oct. 2020, doi:
https://doi.org/10.3390/children71001
82
[21] M. Pominova, E. Kondrateva, M.
Sharaev, A. Bernstein, E. Burnaev,
“Fader networks for domain adaptation
on fmri: abide-ii study,” in 13th Int.
UMT Artificial Intelligence Review
16
Volume 3 Issue 1, Spring 2023