Deep Learning-Assisted MRI Image Segmentation and Classification for Precise Bra
Deep Learning-Assisted MRI Image Segmentation and Classification for Precise Bra
Abstract—For determining the appropriate treatment for brain Automated image segmentation is very important in the
tumors, an accurate diagnosis is necessary. Many studies have medical field because it helps get important information out
focused on the deep learning-based classification of brain tumors. of medical images and helps doctors make diagnoses. Con-
This study employed a comprehensive approach using deep learn-
ing techniques to improve diagnostic accuracy and treatment ventional segmentation takes much time. And is susceptible
planning for brain tumors. The Random Forest algorithm, SVM, to errors that might have an adverse effect on patient care.
YOLOv5, and Neural Network classifier were utilized to classify Therefore, automated approaches to image segmentation are
brain tumors and different planes of brain MRI images. To becoming more widespread [3].
segment the tumors, the U-Net architecture with various neural
network backbones (VGG16, Resnet34, and InceptionV3) was This thesis aims to analyze the use of deep learning al-
implemented, and metrics such as intersection over union (IOU) gorithms for classifying and segmenting brain tumors. Our
and dice coefficient were used to assess the segmentation outcome. goal is to compare different models that are good at finding
The U-Net model achieved an IOU of 0.73 and a dice coefficient and telling the difference between different kinds of brain
of 0.84. The Random Forest algorithm achieved 99 percent and tumors from medical images and also good at separating tumor
95 percent classification accuracy in image planes and tumor
types, respectively. The SVM, YOLOv5, and Neural Network regions. This study aimed to determine how accurately other
Classifier provided classification accuracies of 91 percent, 95 machine learning models classify brain tumors and which
percent, and 96 percent, respectively, in classifying brain tumors. models perform better on our dataset. The objectives of our
The ensemble models predicted the final result, demonstrating study are
the potential of this approach to improve brain tumor diagnosis
and treatment planning. • To classify the tumor types from MRI image
• To segment the tumor from MRI image
Index Terms—Brain tumor, Classification, Machine learning, • To classify the plane of MRI image
Random Forest, U-Net, Semantic segmentation, neural network,
VGG16, Resnet34, InceptionV3, SVM, YOLO. The U-Net architecture is the best deep-learning model for
separating parts of medical images. It is known for its ability to
accurately segment images with high-resolution and detailed
I. I NTRODUCTION structures, such as medical images. The architecture comprises
Brain tumors refer to abnormal growths of cells within the a symmetrical encoder-decoder structure with skip connections
brain or central nervous system and can originate from various between the relevant encoder and decoder layers, making it
types of cells within the brain. Due to their distinct features, easier to localize the segmented item precisely [4]. In our
locations, as well as possible effects on brain function, brain research, we used U-Net with different pre-trained backbones
tumors can be difficult to diagnose and treat [1]. Due to (VGG16, ResNet34, and InceptionV3) as the encoder part
this, more research is being conducted in medical imaging of the architecture [5]–[7]. Using pre-trained backbones can
techniques. Improved imaging techniques have aided in iden- improve the performance of the U-Net by using weights that
tifying and treating brain tumors. However, research for new have been trained on a large dataset. Using different backbones
techniques continues. Early and accurate diagnosis is vital for can also affect the results and help identify the best backbone
optimal treatment and outcomes [2]. for the task at hand. U-Net has been utilized for several
2
Authorized licensed use limited to: Zhejiang University. Downloaded on August 15,2023 at 08:51:50 UTC from IEEE Xplore. Restrictions apply.
D. Inceptionv3 problems. By applying kernel functions to translate the data
The InceptionV3 model is another pre-trained deep learning to a higher-dimensional space where a linear hyperplane may
architecture developed by Google. It belongs to the Inception be utilized to separate the classes, SVM can handle both
model family, which is renowned for its capacity to extract linearly separable and non-linearly separable data. SVM is an
features at different sizes and aspect ratios. The model has 42 established technique for classifying brain tumors in medical
layers and is trained on the ImageNet dataset. The use of” image analysis. SVM has been utilized in the classification of
Inception modules,” blocks of layers that extract information brain tumors to discriminate between various types of tumors
at various sizes using a combination of convolutional and based on their characteristics derived from diagnostic images
pooling layers, is the distinguishing feature of the InceptionV3 like MRI and CT scans. In order to classify the tumor from
architecture. This design helps to reduce the parameters in the the image, SVM has also been employed [9].
model and improve the performance. The InceptionV3 mode
is commonly used as a backbone for variety of computer G. Neural Network Classifier
vision tasks such as image classification, object detection, and
segmentation [7]. Neural network classifiers are machine learning algorithms
When used as a backbone, VGG16, ResNet34, and Incep- that imitate the human brain’s structure and function. They
tionV3 can improve the accuracy of the U-Net model for are composed of interconnected processing nodes that process
image segmentation. VGG16 is known for its ability to input data and produce output predictions. Neurons receive
learn fine- grained features from images and its small filter input signals and generate output signals based on activation
sizes (3x3) which allows it to extract more detailed features. functions. The input layer collects the unprocessed input
ResNet34, due to its residual blocks, can train deep neural data and the output layer generates the final prediction
networks without facing the vanishing gradient problem in an organized neuronal network. Hidden layers process
which allows it to extract deeper features. InceptionV3 is input data and pass their output to the next layer. To
known for its ability to extract features at various scales and lessen the variance between predicted and actual outputs,
aspect ratios which allows it to consider different features training consists modify- ing the connection weights between
from the same image. These features when combined with neurons. Neural networks learn complex patterns, generalize
U-Net architecture, which uses concatenation to add extra to new data, and handle noisy and incomplete data, but
information to the upsampled image, can increase the model’s require significant amounts of training data and can be
overall efficiency. computationally expensive.
3
Authorized licensed use limited to: Zhejiang University. Downloaded on August 15,2023 at 08:51:50 UTC from IEEE Xplore. Restrictions apply.
integrated Conditional Random Field (CRF) with DeepMedic
and a fully Convolutional Neural Network (FCNN). For the
segmentation of brain tumors, the authors of the paper [13] ap-
plied nnU-Net.
Brain tumor classification is another vital task in the
diagnosis process for proper treatment. In their paper [14],
using a hybrid deep learning technique, Asaf Raza et al.
classified glioma, meningioma, and pituitary tumors with
99.67 percent accuracy. In the paper [15], the authors used a
deep convolutional neural network to classify brain tumors.
In the article [16], the authors used pre-trained VGG-16,
ResNet-50, and Inception-V3 and achieved 96 percent, 89
percent, and 75 percent accuracy for the respective pre-trained
Fig. 1. Methodology
weights, while their model provided more precision than
the pr-trained weights. They used MRI images of the brain
in their study. The authors of [17]proposed brain tumors
from MRI images that used a combination of in-depth brain tumors from MRI images, used different backbones to
features and machine learning classifiers. They showed in increase accuracy, and tried to find the best model. We also
their research that a combination of in-depth features might used classification algorithms to classify the image planes and
significantly enhance performance. Also, support vector tumor types.
machines (SVM) with a radial basis function (RBF) kernel Our study aimed to precisely segment the brain tumor
are better than other machine learning classifiers in many in MRI images by employing advanced deep-learning
situations, especially for large datasets. In another study, the techniques. For this purpose, we utilized the U-Net model, a
authors [18] showed how a deep convolutional neural network convolutional neural network architecture. The U-Net model
could automatically segment and classify brain tumors. They was trained on Google Colab using the Adam optimization
used a publicly available MRI dataset collected from 233 algorithm with dice loss and binary cross-entropy [19]. This
patients, and their model obtained 97.3 percent accuracy, was done on a powerful Tesla T4 GPU. The images had a
which was higher than other models used on the same dataset. resolution of 256x256, and we trained the model with 30
epochs and a batch size of 16. Also, we added more U-Net
models with VGG16, ResNet34, and Inceptionv3 as the
IV. DATA backbone architecture on the same platform, with the same
In this study, we used three datasets from Kaggle for our data and parameters, so that we could compare them in the
experiments. The first dataset contained 3064 brain MRI future. We meticulously analyzed the training data of all the
images and their corresponding mask data. We used this models to determine the most suitable model for our dataset.
dataset to train and test our U-Net model. There were 1840 By utilizing such a detailed approach, we aimed to enhance
photos used for training, 640 photos used for validation, and the accuracy of the model’s segmentation capabilities,
640 photos taken for testing. The second dataset contained allowing for more precise brain tumor identification and
over 5800 images, which were classified into four sections: further advancements in medical imaging. To effectively
glioma, meningioma, pituitary tumor, and no tumor. We diagnose and treat a brain tumor, it is essential to know what
trained our random forest, SVM, Neural Network classifier, kind it is. This study focuses on three common types of
and YOLOv5 classifier models with 1321 images of gliomas, brain tumors: gliomas, meningiomas, and pituitary tumors.
1339 images of meningiomas, 1457 images of pituitary In addition, we classified the axial, coronal, and sagittal
tumors, and 1595 images of no tumor. We tested the model planes of MRI images. We trained a model to classify MRI
using 300 images of gliomas, 306 images of meningiomas, image planes using the Random Forest algorithm. To enhance
300 images of pituitary tumors, and 405 images of no tumors. efficiency, we used feature extraction and a 128x128 image
The third dataset contained 3000 images in three different size. Due to the limitations of Google Colab, the image size
planes, coronal, sagittal, and axial, with 1000 images per was restricted to 128x128. We then trained another Random
plane. We used this dataset’s 600 images, 200 images per Forest model to classify brain tumors. We also used the
plane, to test our model. Support Vector Machine algorithm, the YOLOv5 classifier,
and the Neural Network classifier to train three additional
models to classify brain tumors. The same image size and
V. M ETHODOLOGY parameters were used to train all models. The dataset was
Image segmentation and classification are the most basic labeled according to each model’s specifications, and the
steps of image analysis. These two techniques are widely same images were utilized for training, validation, and testing.
used in the medical sector to diagnose different diseases. Finally, to predict the output, the models were ensembled.
In our study, we used the U-Net architecture to segment Image classification performance was evaluated using a set of
4
Authorized licensed use limited to: Zhejiang University. Downloaded on August 15,2023 at 08:51:50 UTC from IEEE Xplore. Restrictions apply.
Accuracy obtained by Classification Algorithms
Model-Name Accuracy Precision Recall
Random Forest 96.87 96.75 96.61
Support Vector Machine 91.07 90.47 90.37
Neural Network Classifier 96.94 96.95 96.94
YOLOv5 Classifier 95.45 95.11 94.39
TABLE II
test images that had not been used during the training phase.
Our approach offers a potentially effective way to enhance the
detection and treatment of brain tumors. When these machine
learning techniques are used well and combined with our
all-around approach, they can help us classify and diagnose
brain tumors more accurately. The whole methodology is
illustrated 1. Fig. 3. Dice Coefficient
VI. R ESULTS AND C OMPARISON without a backbone. Models using VGG16, ResNet34, and
In our study, we used a comprehensive approach, which InceptionV3 all improved the IOU and dice coefficient scores.
included training, validation, and testing of the U-Net models. Figure 4 shows that IOU improved more when the U-Net
In total, we used 1840, 612, and 612 images for training, model was trained with a backbone. However, the Inception
validation, and testing, respectively. Using a backbone, we V3 model had the best performance. Figures 3 and 5 also
observed a significant improvement in the intersection over show a similar trend for the dice coefficient and validation
Union (IOU) and dice coefficient scores, and the Inceptionv3 loss. Additionally, among the classification models, we found
backbone provided the best results. Our findings are presented that the YOLOv5 classifier, Neural Network classifier, and
in Table I, and Figure 2 illustrates the U-Net model’s input Random Forest with feature extraction perform better.
and output. We used the Random Forest algorithm, the Sup-
port Vector Machine, the YOLOv5 classifier, and the Neural
Network classifier to determine the tumors in MRI images. We VII. D ISCUSSION
achieved high accuracy of 99 percent for plane segmentation Image classification and segmentation are crucial in medical
and 91 percent, 95 percent, 95 percent, and 96 percent for the image analysis, especially in diagnosing brain tumors. Ac-
Support Vector Machine, Random Forest, YOLOv5 classifier, curate disease classification is vital in treatment, but limited
and Neural Network classifier, respectively. We did not use screening time can lead to errors. An automated system
any transfer learning techniques for the classification tasks.
The results are presented in Table II. We found that these
models worked better when they were used together and that
using them all together gave more accurate results than using
them one at a time. In conclusion, the results of our study
show that using a combination of machine learning methods
can make brain tumor detection and treatment planning much
more accurate. The results of this study can be beneficial
in understanding how future algorithms for classifying and
segmenting brain tumors will be developed.
Fig. 4. IOU Comparison
Parameters obtained by U-Net models
Model-Name IOU Dice-Coefficient Val. Loss
U-Net 0.6758 0.8046 0.1948
U-Net with vgg16 0.7151 0.8318 0.1663
U-Net with resnet34 0.7298 0.8425 0.1565
U-Net with inceptionv3 0.7323 0.8438 0.1537
TABLE I
5
Authorized licensed use limited to: Zhejiang University. Downloaded on August 15,2023 at 08:51:50 UTC from IEEE Xplore. Restrictions apply.
can assist radiologists in precisely separating the region of clinical utility and impact on patient outcomes, thereby en-
interest, providing a more thorough analysis. This helps hancing the efficiency and accuracy of brain tumor diagnosis
hospitals unable to afford a second radiologist to accurately and treatment.
classify and detect the tumor, improving diagnosis accuracy.
R EFERENCES
Automated systems reduce screening time and enhance the
diagnostic process, providing critical support to radiologists [1] Kenneth Aldape, Kevin M Brindle, Louis Chesler, Rajesh Chopra, Amar
Gajjar, Mark R Gilbert, Nicholas Gottardo, David H Gutmann, Darren
and ensuring more efficient. By adding a backbone, we aimed Hargrave, Eric C Holland, et al. Challenges to curing primary brain
to maximize the performance of the U-net model for medical tumours. Nature reviews Clinical oncology, 16(8):509–520, 2019.
image analysis. We used three datasets obtained from Kaggle [2] Prabhjot Kaur Chahal, Shreelekha Pandey, and Shivani Goel. A survey
on brain tumor detection techniques for mr images. Multimedia Tools
to train U-net models with vgg16, resnet34, and inceptionv3 and Applications, 79(29):21771–21814, 2020.
backbones and compared the results to a U-net model [3] Nahian Siddique, Sidike Paheding, Colin P Elkin, and Vijay Devabhak-
without a backbone. Our results showed that the U-net model tuni. U-net and its variants for medical image segmentation: A review
of theory and applications. Ieee Access, 9:82031–82057, 2021.
trained with the inceptionv3 backbone outperformed the [4] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convo-
other models in terms of the Intersection over Union (IOU) lutional networks for biomedical image segmentation. In International
and Dice coefficient metrics, indicating better segmentation Conference on Medical image computing and computer-assisted inter-
vention, pages 234–241. Springer, 2015.
performance. Furthermore, the inceptionv3 model showed [5] Karen Simonyan and Andrew Zisserman. Very deep convolu-
a more consistent performance and required fewer training tional networks for large-scale image recognition. arXiv preprint
epochs. In brain tumor segmentation, inceptionv3 can be arXiv:1409.1556, 2014.
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep
used as the backbone of U-Net for better performance. residual learning for image recognition. In Proceedings of the IEEE
Additionally, we applied the Random Forest algorithm, conference on computer vision and pattern recognition, pages 770–778,
Support Vector Machine, Neural Network classifier and the 2016.
[7] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed,
YOLOv5 classifier for the classification task of different brain Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew
tumors and MRI image planes. Our results showed that the Rabinovich. Going deeper with convolutions. In Proceedings of the
algorithms performed well in classifying pituitary, glioma, IEEE conference on computer vision and pattern recognition, pages 1–
9, 2015.
and meningioma tumors with an accuracy of maximum 96 [8] Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001.
percent and classifying MRI images into axial, sagittal, and [9] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and
coronal planes with an accuracy of 99 percent. We did not Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems
and their applications, 13(4):18–28, 1998.
use the VGG, ResNet, and inception to train the classification [10] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You
models. Our study highlights the importance of incorporating only look once: Unified, real-time object detection. In Proceedings of
a backbone in the U-Net model. The use of inceptionv3 the IEEE conference on computer vision and pattern recognition, pages
779–788, 2016.
has been shown to improve the model’s performance and [11] Amjad Rehman, Muhammad Attique Khan, Tanzila Saba, Zahid
make the diagnosis process more efficient and accurate. Mehmood, Usman Tariq, and Noor Ayesha. Microscopic brain tumor
However, it is essential to remember that this study’s findings detection and classification using 3d cnn and feature selection architec-
ture. Microscopy Research and Technique, 84(1):133–149, 2021.
are based on a limited amount of data, and further testing [12] Anjali Wadhwa, Anuj Bhardwaj, and Vivek Singh Verma. A review on
with diverse datasets is needed to generalize the results and brain tumor segmentation of mri images. Magnetic resonance imaging,
for practical use in medical diagnosis. Additionally, using 61:247–259, 2019.
[13] Fabian Isensee, Paul F Jäger, Peter M Full, Philipp Vollmuth, and
different classification models together can lead to better Klaus H Maier-Hein. nnu-net for brain tumor segmentation. In
results. International MICCAI Brainlesion Workshop, pages 118–132. Springer,
2021.
[14] Asaf Raza, Huma Ayub, Javed Ali Khan, Ijaz Ahmad, Ahmed S. Salama,
Yousef Ibrahim Daradkeh, Danish Javeed, Ateeq Ur Rehman, and Habib
VIII. C ONCLUSION AND F UTURE WORK Hamam. A hybrid deep learning-based approach for brain tumor
The study used a comprehensive approach to segment brain classification. Electronics, 11(7):1146, 2022.
[15] Zar Nawab Khan Swati, Qinghua Zhao, Muhammad Kabir, Farman Ali,
tumors and classify them by type. U-Net models with different Zakir Ali, Saeed Ahmed, and Jianfeng Lu. Brain tumor classification
backbones were used for segmentation and random forest for mr images using transfer learning and fine-tuning. Computerized
for image plane classification. Various classifiers were used Medical Imaging and Graphics, 75:34–46, 2019.
[16] Hassan Ali Khan, Wu Jue, Muhammad Mushtaq, and Muhammad Umer
and compared, such as SVM, YOLOv5, random forest, and Mushtaq. Brain tumor classification in mri image using convolutional
neural network classifiers. Finally, the models were ensembled neural network. Math. Biosci. Eng, 17(5):6203–6216, 2020.
to predict the output. The research found that the neural [17] Jaeyong Kang, Zahid Ullah, and Jeonghwan Gwak. Mri-based brain
tumor classification using ensemble of deep features and machine
network classifier was the most accurate for classifying the learning classifiers. Sensors, 21(6):2222, 2021.
different tumor types. The study demonstrates the potential [18] Francisco Javier Dı́az-Pernas, Mario Martı́nez-Zarzuela, Mı́riam Antón-
of advanced machine learning techniques in improving brain Rodrı́guez, and David González-Ortega. A deep learning approach
for brain tumor classification and segmentation using a multiscale
tumor diagnosis and treatment accuracy and efficiency. convolutional neural network. In Healthcare, volume 9, page 153.
In the future, your objective is to expand the MRI image MDPI, 2021.
dataset better and more diversely, evaluate the U-net model [19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980, 2014.
with a backbone on CT scans, and conduct extensive testing
on real-world medical data to assess the proposed system’s
6
Authorized licensed use limited to: Zhejiang University. Downloaded on August 15,2023 at 08:51:50 UTC from IEEE Xplore. Restrictions apply.