Deep Learning Based Feature Fusion Model for

Classifying Alzheimer’s Disease Using Brain MRI

Images

Abstract— Worldwide, millions of people suffer from the The stages are mild dementia (MID), moderate dementia
fatal neurological condition known as Alzheimer's disease (MOD), no dementia (ND) and very mild dementia (VMD).
(AD). A prompt identification of AD is essential for effective The combination of the two customized architectures resulted
treatment and better patient outcomes. Deep learning in the extraction of informative features from the brain MRI
techniques have recently become effective tools for images. Using different layer, the pre-trained models have
interpreting medical images and have demonstrated been customized to achieve improved performance compared
promising results in a number of tasks involving disease to various recent studies. The contribution of this work is as
classification. In this paper, a deep learning-based technique follows:
is proposed for classifying Alzheimer's disease using MRI  The paper proposes a CNN that extracts informative
scans. The suggested approach uses a framework that makes features from MRI scans for identifying AD
use of convolutional neural network’s capacity to identify patients.
pertinent characteristics from brain MRI data. First,
preprocessing methods are used to improve image quality and  Two states of art models, namely InceptionV3 and
ResNet50 were utilized in building the framework.
remover noise. A properly curated dataset of brain MRI
images is then used for training and evaluation by combining  The dataset containing 6400 MRI scans, which was
the predictions from InceptionV3 and ResNet50 used in the work had imbalance that was handled
architectures. The proposed technique achieved an AUC of using an oversampling technique known as
84% along with sensitivity of 96%. The work contributes to Synthetic Minority Oversampling Technique
the field of medical image analysis by outlining a successful (SMOTE).
approach based on deep learning for categorizing  The achievement was estimated through accuracy,
Alzheimer’s disease. AUC, precision, recall and F1-score
The following is how the paper is organized: section II
Keywords—Deep Learning, Alzheimer’s disease,
discusses the related works. Section III discusses materials
InceptionV3, ResNet50, MRI
and methods. Section IV presents the proposed methodology.
I. INTRODUCTION Section V describes the experimental results and section VI
concludes the paper along with discussing future work.
Effective therapy and intervention for Alzheimer's disease
depend on early diagnosis. Brain MRI imaging has developed II. BACKGROUND
into a useful technique for AD diagnosis. Deep learning CNNs are ideally suited for spotting early indications of
algorithms have recently made significant strides, and first the disease in AD prediction because they can efficiently take
results in the automatic detection of AD-related patterns from note of the tiny patterns and structural alterations visible in
brain MRI images are encouraging [1].Finding efficient brain MRI images [2].Transfer learning is yet another
therapeutics is among the major difficulties facing AD essential component of deep learning for AD prediction. It is
research. While there isn't a cure for Alzheimer's disease as possible to fine-tune pre-trained CNN models for AD
of yet, a number of treatments and drugs work to control classification tasks, like VGG, ResNet, and Inception, which
symptoms and reduce the disease's growth. were trained on extensive picture datasets like ImageNet.
Transfer learning makes it possible to create precise and
50–80% of dementia patients advance to AD, according to generalizable models even with a small amount of labeled
the National Institutes of Health (NIH) and the Alzheimer's data by transferring knowledge from general picture
Association. Most individuals are identified between the ages attributes to the precise AD prediction job [3]. When
of 65 and 70. However, early diagnosis and prognosis could predicting Alzheimer's disease, ensemble approaches
lessen the impact of sickness emergence. The main challenge combine several models to increase their prediction power.
for Alzheimer's specialists is that there are no reliable AD Combining ResNet50 with Inception-v3 models has showed
medicine available. Nevertheless, modern AD medicines can tremendous performance for predicting AD. Applying
lessen or stop the symptoms' progression. As a result, it is ensemble improves accuracy and effectiveness in AD
critical to spot AD early on. classification by utilizing the advantages of both models and
In this paper, the authors developed an automated deep minimizing each one's specific limitations [4].Persisting
learning framework that is capable of recognizing four stages
challenges in deep learning for Alzheimer’s disease
of AD by combining InceptionV3 and ResNet50 architectures.
prediction include complex model interpretation, limited prediction was the contribution. The potential complexity of
annotated data and class imbalance. Future research aims to the architecture and the requirement for significant computer
address these via interpretable models, domain-driven feature resources were drawbacks.
extraction and diverse datasets for effective training and
evaluation.
Sheng et al. [5] proposed a study, which utilized deep III. MATERIALS AND METHODS
learning, based 3D CNN for early detection of Alzheimer’s The entire system structure for classifying AD using brain
disease. The multiclass classification involved the used of MRI images has been shown in Fig.1.The labelled pictures of
DenseNet. However, generalizability was found to be the MRI dataset were divided into four types. The raw MRI
limited. It is unlikely to perform well on new data or in images were first preprocessed using image preprocessing
different populations. Wang et al. [6] came up with a techniques. The preprocessed images were then divided into a
machine learning framework for predicting the progression training set and a validation set. There was a separate test set
of AD. This study involved hybrid machine learning of of images. First loading the pre-trained architecture of
multiple CNN obtaining an accuracy of 84%. Linear support inceptionV3 and then adding layers on top of it built a
customized inceptionV3 model. In the same way, transfer
vector classifier was utilized. Jain et al. [7] proposed a
learning was applied to the ResNet50 architecture. The
technique, which distinguished between Alzheimer’s disease,
outputs from each of the customized structures were then
Mild Cognitive Impairment (MCI) and Cognitive Normal concatenated and final predictions obtained. The suggested
(CN) using VGG-16. Most significant features had been system’s effectiveness was evaluated using the following
extracted. However, the size of the data was low as there were metrics: confusion matrix, accuracy, AUC, precision, recall
50 samples for each category. Waleed Al Shehri [8] classified and F1-score. The entire system was built using Python.
Alzheimer’s disease into four different categories by using
DenseNet169 and ResNet50 architectures. The work showed A. Alzheimer’s disease (AD) MRI dataset
that DenseNet169 performed better. Early detection of AD The current work makes use of the publicly
stages using multifractal geometry analysis [9] involved the accessible MRI image collection on Kaggle [16].There are
use of KNN. Two different datasets were used in the study. four different types of AD MRI scans included in it: ND (non-
The classification had been carried out through multifractal demented), MID (mildly demented), and VMD (very mildly
geometry. Lanjewar et al. [10] proposed a framework that demented). The dataset consists of 6400 images of the four
combined CNN with KNN for detecting AD using MRI different categories of AD as illustrated in in Fig. 2a and 2b.
images. In this paper, KNN was applied to extracted features The information is gathered from a number of websites,
from CNN. ResNet50, VGG16 and MobileNetV2 hospitals, and public databases. This dataset is unbalanced.
architectures were employed for detecting AD and their There are many levels of AD severity. The patient starts to
performances were compared with the CNN-KNN model. forget throughout the VMD period. The patient starts to forget
Another study [11] proposed a deep learning based AD things such as people’s names and the locations of their
diagnosis using multi-modal MRI data. The work used belongings. Utilizing cognitive ability assessments is
structural and functional data from multi-modal MRI scans challenging. In the MID phase, patients lose focus and their
with a deep learning approach to diagnose AD. The model, ability to perform tasks, cannot find their way to the desired
which outperformed single-modal models in accuracy, made area, and cannot recollect what they were saying. Patients
use of a multi-modal CNN architecture. The contribution was might not even be aware of memory loss at this time.
the better categorization of AD by the integration of Cognitive testing can be used at this point to find out. During
multimodal data. Due to the variety of the data sources, the MD phase, the patient starts to forget recent
substantial preprocessing was necessary, which was a
restriction. Another strategy [12] contributed in early
detection of AD by using feature fusion techniques as well as
presented a group of CNN architectures. The use of ensemble
methods had resulted in improved accuracy. Potential
processing overhead and the requirement for effective model
selection were limitations. Qiu et al. [13] suggested an
interpretable deep learning strategy for AD categorization.
The model used integrated gradients to draw attention to key
areas of the images that affected categorization choices. The
emphasis on model interpretability was the contribution. The
potential trade-off between accuracy and interpretability was
a constraint. In the paper by M. Zaabi et al. [14], transfer
learning for CNN-based AD classification was investigated.
For feature extraction, it used pre-trained algorithms on large
image datasets. The adaption of transfer learning to AD
prediction was the contribution. Limitations included the
requirement for domain adaptation because general and
medical images varied from one another. Another work [15]
concentrated on using brain MRI volumes and 3D CNNs to Fig. 1. The overall system framework of the proposed AD categorization
classify AD. The MRI data's spatial patterns were accurately system
represented by the model. The use of 3D CNNs for AD
accomplishments and significant past events, struggles with C. Feature fusion using convolutional neural network
cost calculations, finds it difficult to travel alone, and loses (CNN)
empathy. One of the pre-trained models that was employed in the
work was Inception V3. This CNN architecture contains
characteristics, which make it computationally efficient along
with high efficiency, thus, making it a suitable choice for the
overall model. 32 different layers were added on top of the
pre-trained InceptionV3 for customization.
The other architecture that was used in the paper was
ResNet50. A total of 21 layers were added on top of the pre-
trained ResNet50 architecture. The predictions from each of
the customized structures were then concatenated. By
concatenating the outputs, the authors have fused the features
learnt from each structure with the intent of enhancing the
model’s ability to capture complex patterns from the data. The
two customized structures captured different aspects of the
data. Their combination through concatenation introduced
diversity in the model as it gained the ability to generalize
(a) better across variations of the input data. Fig. 3 shows the
overall structure of the model.

MID MOD

ND VMD Fig. 3. Classification model structure

(b) D. Deep Convolutional Neural Networks

In this paper, two deep convolutional neural networks
Fig. 2. a Class-wise distribution of total MRI images in the dataset and b
dataset images of four classes were employed to classify AD.
Residual Neural Network (ResNet): ResNet makes use of
an innovative design to operate many more convolutional
B. Pre-processing layers without encountering more difficulties [17, 18, 19]. In
The pre-processing stage consisted of two operations. the work, the pre-trained ResNet50 model was loaded and
They are : image augmentation and application of Synthetic custom layers were added on top. A dropout layer of dropout
Minority Oversampling technique (SMOTE).The rate 0.5 was added. Then a flatten layer was added followed
transformations involved in the data augmentation were by a BatchNormalization layer. A series of dense layers with
zooming, brightness adjustment, flipping of image number of units 1024, 512 and 256 were applied with dropout
horizontally, filling of empty regions created due to zooming
rates as 0.5. The activation functions included were ‘ReLu’.
with a constant value, specifying the ordering of the
dimensions of the image data. The image augmentation was
followed by the application of SMOTE. The dataset is an InceptionV3: The proposed work made use of a customized
imbalanced one. There are 64 samples of the mild-demented inceptionV3 model by adding custom layers on top. Flatten,
category while 3200 samples for no dementia. This indicates dropout, batch-normalization and dense layers were added
the imbalance present which was therefore, handled by using along with activation function. The model was intended to
SMOTE. classify images into 4 classes based on the architecture. A
dropout layer with a dropout rate of 0.5 was added first. It
helps preventing overfitting by randomly dropping out a
fraction of neurons during the training. Then a score, precision and recall and sensitivity. Each of them are
GlobalAveragePooling layer was added to calculate the significant in determining how well the model performed in
average of each feature map across all spatial positions. It classifying the four different levels of severity of Alzheimer’s
reduces the spatial dimensions of the features. Then a flatten disease.
layer was added for converting the 2D feature maps into a 1D
vector. Then a batch normalization layer was added. A series
of dense layers were added that included units of V. EXPERIMENTAL RESULTS AND DISCUSSION
256,128,64,64,32,32 and 4.The final consisted of 4 neurons
because this represents the number of output classes, which This section elaborates on the results obtained and
is four. For each dense layer, a dropout of rate 0.5 was added analysis of the performance of the proposed model. Table I
and the activation method employed was ‘ReLU’. shows the statistical results obtained indicating how the
proposed model performed when tested on the test data. It is
IV. PROPOSED METHODOLOGY observed that the proposed work achieved an AUC of 84%
Fig.1. presents the study’s research methodology in order to along with sensitivity of 96%.
find a solution to AD’s early diagnosis.
TABLE I. OVERALL PERFORMANCE OF MODEL
A. Model building
The proposed model was based on the InceptionV3 and Evaluation Metrics
ResNet50 models. Convolutional Neural Network is a deep
AUC Precision Recall F1-score Sensitivity
learning model that has its use in classifying features. A CNN
model composes of several different layers, which have
specific roles in the classification task. A few common layers 0.84 0.56 0.56 0.56 0.96
which had been added to the pre-trained architectures are the
input layer, batch normalization layer, ‘ReLU’ activation
layer, fully connected layer , softmax layer and the
The training results can be seen through the graph from
classification layer. The input layer of the first CNN layer
specified the 112* 112 * 3 (width * height * channels) picture Fig. 4, which consists of the variation of the training
size that was utilized in the work. accuracy, AUC and loss with epochs.

classification is provided by the topmost layer. Equal to the

Model training
After the model had been built and compiled, training of
the model on the dataset was carried out. 5121 images were
used for training while 1279 images were used for testing.
This means that 80% of all the scans in the dataset were
utilized for training while the rest 20% was used for testing
the model.
B. Exporting trained model
The trained models were exported after training for
testing and additional use. There is no need to repeatedly train
once the entire dataset has been used for training. Further
processing is done using the trained models.
C. Model testing
Several images were used to test the trained model. This
methodology diagnoses AD and categorizes it into these four
groups. The model's output values are contrasted with the
actual values of the input photos. The model is assessed using
the comparison of output values with true values for the
testing dataset.

D. Model evaluation
Based on the training model and testing dataset, the model
is assessed. By comparing calculated values from the model
and true values known for each image in the testing dataset,
the evaluation measure is derived. The effectiveness of the
model is demonstrated by the evaluation measures. The
evaluation metric used to determine the importance of the
model is accuracy. The proposed model's performance
analysis makes use of Area Under ROC curve (AUC), F1-
Fig. 4. Progress of evaluation metrics during training
 The confusion matrix of the proposed model is shown in VI. CONCLUSION AND FUTURE WORK
Fig.5 Alzheimer's disease significantly influences people,
families, and healthcare systems on an emotional, physical,
and monetary level. The burden of Alzheimer's disease on
aging populations is growing as life expectancies rise around
the world, emphasizing the significance of early diagnosis,
efficient caregiving, and continued research efforts to
develop improved treatments and ultimately a cure. The
ability of detecting and identifying early-stage AD in the
healthcare industry always remains tricky. In this study, we
presented a framework based on two convolutional neural
network architectures, such as InceptionV3 and ResNet50,
where their individual outputs have been concatenated in
order to produce the final prediction. The classification
method has been constructed in a way in which the powerful
capabilities of the two structures of these models have been
utilized in order to produce accurate predictions. In this study,
which employed brain MRI images to classify Alzheimer's
Fig. 5. Confusion matrix disease, the authors have used a deep learning based model
It acts as a useful measure in understanding the predictions of concatenation technique that harnessed the strength of the
the model better. It indicates how the model is performing in ResNet-50 and Inception v3 architectures. The objective was
predicting each class. to improve illness categorization performance and accuracy.
Four test cases were considered for testing the model on The testing outcomes demonstrated that the effectiveness
individual images of different classes. Fig. 6 shows test case of our strategy was quite promising. A crucial diagnostic
1, where classification is performed for the “Very Mild indicator called precision received a score of 56%. This
Demented” class and the proposed model classified shows that the model can accurately detect positive cases
accurately with 72.45% probability. Fig. 7 shows test case 2 while reducing false positives. In addition, the recall value,
for testing the “Mild Demented” class and the trained model which gauges how well the model can recognize all positive
classified it successfully, where the probability of the cases, was 56%. This shows how sensitive the model is to real
identified classification was 98.6%. Fig. 8 shows the test case positive examples. A value of 84% was achieved by the area
3 results for the “Moderate Demented” test sample data and under the ROC curve (AUC), a commonly utilized indicator
the proposed model successfully predicted the classification of classification performance. This shows that the model can
with 100.00% probability. Lastly, Fig. 9 shows test case 4 for distinguish between positive and negative cases with almost
testing the “No dementia” class and the trained model perfect precision.
classified it successfully with an accuracy of 96.91%. Finally, the F1-score, which balances recall and precision,
came in at 56%. The model's capacity to produce a pleasing
balance of precise positive identifications and few erroneous
negatives is highlighted by this.
The efficacy of our suggested method in classifying
Alzheimer's disease is highlighted by the performance
achieved across all of these parameters. Inception v3 and
ResNet-50 can be combined to produce precise and
dependable classification results in medical image analysis,
as seen by the produced accuracy, precision, recall, AUC, and
F1-score.These findings provide a foundation for employing
cutting-edge deep learning techniques to better Alzheimer's
Fig. 6. Test case 1 Fig. 7. Test case 2 disease diagnosis and monitoring.
While our study achieved satisfactory outcomes, avenues for
further development remain ripe. The following are potential
areas that can be explored to build upon our research:
 Using a larger and varied dataset from the archives
of the Open Access Series of Imaging Studies
(OASIS) or the Alzheimer's Disease Neuroimaging
Initiative (ADNI) as the dataset used in the study
was of limited size
 The extracted features from a CNN can be given to
the Machine Learning classifiers, such as SVM or
Random Forest (RF).
 Incorporating multiple modalities such as PET scans
along with MRI scans in order to produce a more
Fig. 8. Test case 3 Fig. 9. Test case 4
reliable system
  Using techniques other than SMOTE in order to analysis based on k-nearest-neighbour algorithm,” scientific reports,
12:22381,2022.
avoid the negative impacts of it
[10] M.G. Lanjewar, J.S. Pranab and A.Y. Shaikh,“Development of
 Observing the performance along with the framework by combining CNN with KNN to detect Alzheimer disease
computational complexity through deploying using MRI images,” Multimedia Tools and Applications, 82:12699-
Vision Transformer (ViT). 12717,2022.
[11] G. Lee, K. Nho, B. Kang, K.-A. Sohn, and D. Kim, "Predicting
REFERENCES Alzheimer’s disease progression using multi-modal deep learning
approach," Sci. Rep., vol. 9, no. 1, p. 1952, 2019
[12] A. Loddo, S. Buttau, and C. Di Ruberto, "Deep learning based pipelines
[1] J. Ebersole, G. Rose, T. Eid, K. Behar, and P. Patrylo, "Altered for Alzheimer's disease diagnosis: a comparative study and a novel
hippocampal astroglial metabolism is associated with aging and deep-ensemble method," Computers in Biology and Medicine, vol.
preserved spatial learning and memory," Neurobiology of Aging, vol. 141, p. 105032, 2022.
102, pp. 188-199, 2021. DOI: 10.1016/j.neurobiolaging.2021.02.015. [13] S. Qiu, P. S. Joshi, M. I. Miller, C. Xue, X. Zhou, C. Karjadi, G. H.
[2] D. Shen, G. Wu, and H.-I. Suk, "Deep Learning in Medical Image Chang et al., "Development and validation of an interpretable deep
Analysis," Annual Review of Biomedical Engineering, vol. 19, no. 1, learning framework for Alzheimer’s disease classification," Brain, vol.
pp. 221-248, 2017. DOI: 10.1146/annurev-bioeng-071516-044442. 143, no. 6, pp. 1920-1933, 2020.
[3] A. Esteva, A. Robicquet, B. Ramsundar, V. Kuleshov, M. DePristo, K. [14] M. Zaabi, N. Smaoui, H. Derbel, and W. Hariri, "Alzheimer's disease
Chou, et al., "A guide to deep learning in healthcare," Nature Medicine, detection using convolutional neural networks and transfer learning
vol. 25, no. 1, pp. 24-29, 2019. DOI: 10.1038/s41591-018-0316-z. based methods," in 2020 17th International Multi-Conference on
[4] P. M. Burlina, N. J. Joshi, et al., "AI-based detection of erythema Systems, Signals & Devices (SSD), pp. 939-943, IEEE, 2020.
migrans and disambiguation against other skin lesions.” Computers in [15] A. Farooq, S. Anwar, M. Awais, and S. Rehman, "A deep CNN based
Biology and Medicine, 103977, 2020. DOI: multi-class classification of Alzheimer’s disease using MRI," 2017
10.1016/j.compbiomed.2020.103977. IEEE International Conference on Imaging Systems and Techniques
[5] Sheng, Arjun, Henry & Jingyun. “Generalizable deep learning model (IST), 2017. DOI: 10.1109/ist.2017.8261460
for early alzheimer’s disease detection from structural MRIs,” [16] Alzheimer's Dataset (4 class of Images), 26 December,2019.Kaggle
scientific reports.,12:17106,2022. [Online].Available:https://www.kaggle.com/datasets/tourist55/alzhei
[6] C. Wang, Y. Li, Y. Tsuboshita, et al., "A high-generalizability machine mers-dataset-4-class-of-images
learning framework for predicting the progression of Alzheimer’s [17] Lanjewar MG, Gurav OL.”Convolutional neural networks based
disease using limited data," npj Digital Medicine, vol. 5, no. 1, p. 43, classifications of soil images.”Multimed Tools Appl 81:10313–10336.
2022. DOI: 10.1038/s41746-022-00577-x. https://doi.org/10.1007/s11042-022-12200-y,2022
[7] R.Jain, N.Jain, A. Aggarwal and D.J. Hemanth, “Convolutional neural [18] Lanjewar MG, Morajkar PP, Parab J. “Detection of tartrazine colored
network based alzheimer’s disease classification from magnetic rice flour adulteration in turmeric from multi-spectral images on
resonance brain images,” Cognitive Systems Research. 57:147-159, smartphone using convolutional neural network deployed on PaaS
2019. cloud.”Multimed Tools Appl 81:16537–16562.
[8] Waleed Al Shehri. “Alzheimer’s disease diagnosis and classification https://doi.org/10.1007/s11042-022-12392-3,2022.
using deep learning techniques,” PeerJ Computer Science, 8:e1177, [19] Understanding Deep Convolutional Neural Networks.(n.d)
2022. https://www.run.ai/guides/deep-learning-for-computer-vision/deep-
[9] Elgammal, Zahran and M. Addelsalam. “A new strategy for the early convolutional-neural-networks. [Accessed on 26 August 2023]
detection of alzheimer disease stages using multifractal geometry