A Novel Multi-Feature Fusion Method for Classification of Gastrointestinal Diseases Using Endoscopy Images
Abstract
:1. Introduction
2. Related Work
2.1. Machine Learning Methods
2.2. Deep Learning Methods
2.3. Research Gaps and Motivation
- Most of the research works for the classification of GI diseases were conducted with limited datasets due to privacy concerns and the rarity of abnormalities. Hence, to improve the effectiveness of these models, these methods must be analysed with benchmark datasets.
- Though the cumulative performance of the above-discussed methods were considerable, the class-wise metrics are often overlooked. This is due to the similarity in the morphological features existing between two or more diseases. Identification of precise hand-crafted features is a challenging task. Hence, the power of deep learning methods needs to exploited in the place of machine learning methods, which require manual feature extraction.
- Though few deep learning research works were already reported for GI disease classification, these methods were restricted to pre-trained models or a fusion of pre-trained models. Hence, there exists a vital need to apply suitable architectural enhancements/customisations to these models for improved performance.
- The proposed experiments were conducted with the HyperKvasir benchmark dataset for better generalisation of all classes. This ensures that the proposed method is evaluated for a benchmark dataset with 23 different classes of GI.
- The proposed research presents a hybrid deep learning approach involving a pre-trained network and a custom CNN. While EfficientNet B0 was applied on one track to extract prominent features, custom CNN was employed on the other side for effective feature calibration. Finally, the features from both networks are fused to represent a high context feature vector representing each class.
- We have proposed an effective feature extractor namely ‘Effimix’. It involved the application of squeeze and excitation layers, background elimination, and a non-monotonic activation function. By combining the features from Effimix and EfficientNet B0 backbone, the proposed feature fusion network was able to achieve good inter-class metrics.
3. Proposed System
3.1. Dataset Description
3.2. Data Augmentation
3.3. EfficientNet B0
3.4. Effimix
3.5. Classification
4. Results and Discussion
4.1. Environmental Setup
4.2. Ablation Studies
4.2.1. Analysis of the EfficientNet B0 Network
4.2.2. Analysis of the Effimix Network
4.2.3. Analysis of the Proposed Feature Fusion Network
4.3. Performance Analysis
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Muthusamy, V.R.; Lightdale, J.R.; Acosta, R.D.; Chandrasekhara, V.; Chathadi, K.V.; Eloubeidi, M.A.; Fanelli, R.D.; Fonkalsrud, L.; Faulx, A.L.; Khashab, M.A.; et al. The Role of Endoscopy in the Management of GERD. Gastrointest Endosc 2015, 81, 1305–1310. [Google Scholar] [CrossRef] [PubMed]
- Galdran, A.; Carneiro, G.; Ballester, M.A.G. A Hierarchical Multi-task Approach to Gastrointestinal Image Analysis. Lect. Notes Comput. Sci. 2021, 12668, 275–282, (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). [Google Scholar] [CrossRef]
- Sivakumar, P.; Kumar, B.M. A novel method to detect bleeding frame and region in wireless capsule endoscopy video. Clust. Comput. 2018, 22, 12219–12225. [Google Scholar] [CrossRef]
- Hassan, A.R.; Haque, M.A. Computer-aided gastrointestinal hemorrhage detection in wireless capsule endoscopy videos. Comput. Methods Programs Biomed. 2015, 122, 341–353. [Google Scholar] [CrossRef]
- Ali, H.; Yasmin, M.; Sharif, M.; Rehmani, M.H. Computer assisted gastric abnormalities detection using hybrid texture descriptors for chromoendoscopy images. Comput. Methods Programs Biomed. 2018, 157, 39–47. [Google Scholar] [CrossRef] [PubMed]
- Jani, K.; Srivastava, R.; Srivastava, S. Computer Aided Medical Image Analysis for Capsule Endoscopy using Multi-class Classifier. In Proceedings of the 2019 IEEE 5th International Conference for Convergence in Technology, I2CT 2019, Bombay, India, 29–31 March 2019. [Google Scholar] [CrossRef]
- Charfi, S.; Ansari, M.E. Computer-aided diagnosis system for colon abnormalities detection in wireless capsule endoscopy images. Multimed. Tools Appl. 2017, 77, 4047–4064. [Google Scholar] [CrossRef]
- Moccia, S.; Vanone, G.O.; De Momi, E.; Laborai, A.; Guastini, L.; Peretti, G.; Mattos, L.S. Learning-based classification of informative laryngoscopic frames. Comput. Methods Programs Biomed. 2018, 158, 21–30. [Google Scholar] [CrossRef]
- de Souza, L.A.; Afonso, L.; Ebigbo, A.; Probst, A.; Messmann, H.; Mendel, R.; Hook, C.; Palm, C.; Papa, J.P. Learning visual representations with optimum-path forest and its applications to Barrett’s esophagus and adenocarcinoma diagnosis. Neural Comput. Appl. 2019, 32, 759–775. [Google Scholar] [CrossRef]
- Al-Rajab, M.; Lu, J.; Xu, Q. Examining applying high performance genetic data feature selection and classification algorithms for colon cancer diagnosis. Comput. Methods Programs Biomed. 2017, 146, 11–24. [Google Scholar] [CrossRef]
- Igarashi, S.; Sasaki, Y.; Mikami, T.; Sakuraba, H.; Fukuda, S. Anatomical classification of upper gastrointestinal organs under various image capture conditions using AlexNet. Comput. Biol. Med. 2020, 124, 103950. [Google Scholar] [CrossRef]
- KahsayGebreslassie, A.; Hagos, M.T. Automated Gastrointestinal Disease Recognition for Endoscopic Images. In Proceedings of the 2019 International Conference on Computing, Communication, and Intelligent Systems, ICCCIS 2019, Greater Noida, India, 18–19 October 2019; pp. 312–316. [Google Scholar] [CrossRef]
- Qiu, W.; Xie, J.; Shen, Y.; Xu, J.; Liang, J. Endoscopic image recognition method of gastric cancer based on deep learning model. Expert Syst. 2022, 39, e12758. [Google Scholar] [CrossRef]
- Öztürk, Ş.; Özkaya, U. Residual LSTM layered CNN for classification of gastrointestinal tract diseases. J. Biomed. Inform. 2021, 113, 103638. [Google Scholar] [CrossRef] [PubMed]
- Valério, M.T.; Gomes, S.; Salgado, M.; Oliveira, H.P.; Cunha, A. Lesions Multiclass Classification in Endoscopic Capsule Frames. Procedia Comput. Sci. 2019, 164, 637–645. [Google Scholar] [CrossRef]
- Rahman, M.M.; Wadud, M.A.H.; Hasan, M.M. Computerized classification of gastrointestinal polyps using stacking ensemble of convolutional neural network. Inform. Med. Unlocked 2021, 24, 100603. [Google Scholar] [CrossRef]
- Zeng, F.; Li, X.; Deng, X.; Yao, L.; Lian, G. An image classification model based on transfer learning for ulcerative proctitis. Multimed. Syst. 2021, 27, 627–636. [Google Scholar] [CrossRef]
- Ellahyani, A.; Jaafari, I.E.; Charfi, S.; Ansari, M.E. Fine-tuned deep neural networks for polyp detection in colonoscopy images. Pers. Ubiquitous Comput. 2022, 1–13. [Google Scholar] [CrossRef]
- Lafraxo, S.; Ansari, M.E. GastroNet: Abnormalities Recognition in Gastrointestinal Tract through Endoscopic Imagery using Deep Learning Techniques. In Proceedings of the 2020 8th International Conference on Wireless Networks and Mobile Communications (WINCOM), Reims, France, 27–29 October 2020. [Google Scholar] [CrossRef]
- He, Q.; Bano, S.; Stoyanov, D.; Zuo, S. Hybrid Loss with Network Trimming for Disease Recognition in Gastrointestinal Endoscopy. Lect. Notes Comput. Sci. 2021, 12668, 299–306, (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). [Google Scholar] [CrossRef]
- Galdran, A.; Carneiro, G.; Ballester, M.A.G. Balanced-MixUp for Highly Imbalanced Medical Image Classification. Lect. Notes Comput. Sci. 2021, 12905, 323–333, (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). [Google Scholar] [CrossRef]
- Barbhuiya, F.A.; Dutta, A.; Bhattacharjee, R.K. Efficient Detection of Lesions During Endoscopy. Available online: https://link.springer.com/chapter/10.1007/978-3-030-68793-9_24 (accessed on 7 August 2022).
- Zhao, Q.; Yang, W.; Liao, Q. AFA-RN: An Abnormal Feature Attention Relation Network for Multi-class Disease Classification in gastrointestinal endoscopic images. In Proceedings of the BHI 2021–2021 IEEE EMBS International Conference on Biomedical and Health Informatics, Athens, Greece, 27–30 July 2021. [Google Scholar] [CrossRef]
- Luo, X.; Zhang, J.; Li, Z.; Yang, R. Diagnosis of ulcerative colitis from endoscopic images based on deep learning. Biomed. Signal Process. Control 2022, 73, 103443. [Google Scholar] [CrossRef]
- Wang, Z.; Li, Z.; Xiao, Y.; Liu, X.; Hou, M.; Chen, S. Three feature streams based on a convolutional neural network for early esophageal cancer identification. Multimed. Tools Appl. 2022, 1–18. [Google Scholar] [CrossRef]
- Iakovidis, D.K.; Georgakopoulos, S.V.; Vasilakakis, M.; Koulaouzidis, A.; Plagianakos, V.P. Detecting and Locating Gastrointestinal Anomalies Using Deep Learning and Iterative Cluster Unification. IEEE Trans. Med. Imaging 2018, 37, 2196–2210. [Google Scholar] [CrossRef] [PubMed]
- Cao, J.; Yao, J.; Zhang, Z.; Cheng, S.; Li, S.; Zhu, J.; He, X.; Jiang, Q. EFAG-CNN: Effectively fused attention guided convolutional neural network for WCE image classification. In Proceedings of the 2021 IEEE 10th Data Driven Control and Learning Systems Conference, DDCLS 2021, Suzhou, China, 14–16 May 2021; pp. 66–71. [Google Scholar] [CrossRef]
- Rahim, T.; Hassan, S.A.; Shin, S.Y. A deep convolutional neural network for the detection of polyps in colonoscopy images. Biomed. Signal Process. Control 2021, 68, 102654. [Google Scholar] [CrossRef]
- Hatami, S.; Shamsaee, R.; Olyaei, M.H. Detection and classification of gastric precancerous diseases using deep learning. In Proceedings of the 6th Iranian Conference on Signal Processing and Intelligent Systems, ICSPIS 2020, Mashhad, Iran, 23–24 December 2020. [Google Scholar] [CrossRef]
- Gjestang, H.L.; Hicks, S.A.; Thambawita, V.; Halvorsen, P.; Riegler, M.A. A self-learning teacher-student framework for gastrointestinal image classification. In Proceedings of the IEEE Symposium on Computer-Based Medical Systems, Aveiro, Portugal, 7–9 June 2021; pp. 539–544. [Google Scholar] [CrossRef]
- Jin, Y.; Hu, Y.; Jiang, Z.; Zheng, Q. Polyp segmentation with convolutional MLP. Vis. Comput. 2022, 1–19. [Google Scholar] [CrossRef]
- Ji, G.-P.; Xiao, G.; Chou, Y.-C.; Fan, D.-P.; Zhao, K.; Chen, G.; Van Gool, L. Video Polyp Segmentation: A Deep Learning Perspective (Version 3). arXiv 2022, arXiv:2203.14291. [Google Scholar]
- Zhang, R.; Li, G.; Li, Z.; Cui, S.; Qian, D.; Yu, Y. Adaptive Context Selection for Polyp Segmentation. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2020; MICCAI 2020. Lecture Notes in Computer Science; Springer: Cham, Germany, 2020; Volume 12266. [Google Scholar] [CrossRef]
- Borgli, H.; Thambawita, V.; Smedsrud, P.H.; Hicks, S.; Jha, D.; Eskeland, S.L.; Randel, K.R.; Pogorelov, K.; Lux, M.; Nguyen, D.T.D.; et al. HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci. Data 2020, 7, 283. [Google Scholar] [CrossRef]
- Tan, M.; Le, Q.V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Available online: http://proceedings.mlr.press/v97/tan19a.html (accessed on 7 August 2022).
- Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.-C. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4510–4520. [Google Scholar] [CrossRef]
- Zhu, M.; Chen, Z.; Yuan, Y. DSI-Net: Deep Synergistic Interaction Network for Joint Classification and Segmentation With Endoscope Images. IEEE Trans. Med. Imaging 2021, 40, 3315–3325. [Google Scholar] [CrossRef]
- Misra, D. Mish: A Self-regularized Non-Monotonic Neural Activation Function. arXiv 2019, arXiv:1908.08681. [Google Scholar]
Sl. No. | Class Name | No. of Samples |
---|---|---|
1 | barretts-short | 53 |
2 | barretts | 41 |
3 | bbps-0-1 | 646 |
4 | bbps-2-3 | 1148 |
5 | cecum | 1009 |
6 | dyed-lifted-polyps | 1002 |
7 | dyed-resection-margins | 989 |
8 | esophagitis-a | 403 |
9 | esophagitis-b-d | 260 |
10 | hemorrhoids | 6 |
11 | ileum | 9 |
12 | impacted-stool | 131 |
13 | polyps | 1028 |
14 | pylorus | 999 |
15 | retroflex-rectum | 391 |
16 | retroflex-stomach | 764 |
17 | ulcerative-colitis-grade-0-1 | 35 |
18 | ulcerative-colitis-grade-1-2 | 11 |
19 | ulcerative-colitis-grade-1_ | 201 |
20 | ulcerative-colitis-grade-2-3 | 28 |
21 | ulcerative-colitis-grade-2 | 443 |
22 | ulcerative-colitis-grade-3 | 133 |
23 | z-line | 932 |
Sl. No. | Model | Accuracy | Macro Average | Weighted Average | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
F1 | Precision | Recall | MCC | Kappa | F1 | Precision | Recall | MCC | Kappa | |||
1 | EfficientNet B0 | 0.956 | 0.95 | 0.95 | 0.95 | 0.954 | 0.956 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 |
2 | Effimix Network | 0.854 | 0.85 | 0.85 | 0.85 | 0.85 | 0.856 | 0.85 | 0.85 | 0.85 | 0.85 | 0.856 |
3 | Proposed network | 0.9799 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.97 | 0.98 | 0.97 | 0.98 | 0.98 |
S. No. | Class Name | EfficientNet B0 | Effimix | Combined Model | ||||||
---|---|---|---|---|---|---|---|---|---|---|
F1 | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | ||
1 | barretts-short | 0.941 | 0.932 | 0.949 | 0.64 | 0.71 | 0.58 | 0.96 | 0.97 | 0.95 |
2 | barretts | 0.937 | 0.91 | 0.95 | 0.67 | 0.77 | 0.58 | 0.99 | 0.94 | 0.97 |
3 | bbps-0-1 | 0.97 | 0.97 | 0.98 | 0.98 | 0.99 | 0.97 | 0.99 | 1 | 0.99 |
4 | bbps-2-3 | 0.99 | 0.99 | 1 | 1 | 1 | 1 | 0.99 | 1 | 0.99 |
5 | cecum | 0.97 | 1 | 0.94 | 0.99 | 0.99 | 0.99 | 1 | 1 | 1 |
6 | dyed-lifted-polyps | 0.95 | 0.97 | 0.94 | 0.77 | 0.84 | 0.81 | 0.96 | 1 | 0.94 |
7 | dyed-resection-margins | 0.96 | 0.95 | 0.97 | 0.83 | 0.78 | 0.88 | 0.96 | 0.94 | 1 |
8 | esophagitis-a | 0.85 | 0.87 | 0.83 | 0.44 | 0.45 | 0.43 | 0.92 | 0.92 | 0.91 |
9 | esophagitis-b-d | 0.93 | 0.97 | 0.89 | 0.69 | 0.65 | 0.74 | 0.96 | 0.96 | 0.97 |
10 | hemorrhoids | 1 | 1 | 1 | 0.98 | 0.97 | 1 | 1 | 1 | 1 |
11 | ileum | 0.99 | 0.99 | 1 | 0.99 | 0.98 | 1 | 1 | 1 | 1 |
12 | impacted-stool | 0.99 | 0.99 | 0.98 | 0.99 | 0.99 | 1 | 0.99 | 0.99 | 1 |
13 | polyps | 1 | 1 | 1 | 0.98 | 0.99 | 0.98 | 0.99 | 1 | 0.99 |
14 | pylorus | 0.96 | 0.93 | 1 | 0.98 | 0.99 | 0.97 | 0.99 | 0.99 | 1 |
15 | retroflex-rectum | 0.98 | 0.99 | 0.97 | 0.94 | 0.97 | 0.91 | 0.99 | 0.99 | 0.98 |
16 | retroflex-stomach | 0.98 | 0.99 | 0.98 | 0.92 | 0.89 | 0.95 | 0.99 | 0.99 | 0.99 |
17 | ulcerative-colitis-grade-0-1 | 0.93 | 0.88 | 0.99 | 0.94 | 0.91 | 0.96 | 0.98 | 0.96 | 1 |
18 | ulcerative-colitis-grade-1-2 | 1 | 1 | 1 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 1 |
19 | ulcerative-colitis-grade-1_ | 0.91 | 0.94 | 0.88 | 0.72 | 0.78 | 0.67 | 0.95 | 0.99 | 0.91 |
20 | ulcerative-colitis-grade-2-3 | 0.97 | 1 | 0.95 | 0.94 | 0.89 | 0.99 | 0.99 | 0.98 | 1 |
21 | ulcerative-colitis-grade-2 | 0.87 | 0.9 | 0.84 | 0.66 | 0.7 | 0.63 | 0.95 | 0.96 | 0.94 |
22 | ulcerative-colitis-grade-3 | 0.95 | 0.92 | 0.98 | 0.86 | 0.8 | 0.93 | 0.96 | 0.92 | 1 |
23 | z-line | 0.88 | 0.84 | 0.92 | 0.6 | 0.54 | 0.68 | 0.95 | 0.96 | 0.94 |
Average | 0.95 | 0.95 | 0.95 | 0.84 | 0.85 | 0.85 | 0.97 | 0.97 | 0.97 |
S. No. | Network | Hyper Parameters | Total Number of Trainable Parameters |
---|---|---|---|
1 | EfficientNet B0 | Optimiser: Adam optimiser No. of epochs: 50 Learning rate: 0.001 Weight decay: 0.0001 | 1,729,176 |
2 | Effimix Network | Optimiser: Adam optimiser No. of epochs: 50 Learning rate: 0.0001 Weight decay: 0.0001 | 5,288,548 |
3 | Proposed feature fusion network | Optimiser: Adam optimiser No. of epochs: 100 Learning rate: 0.0001 Weight decay: 0.0001 | 7,041,276 |
S. No. | Source | Method | F1 | Precision | Accuracy | Recall |
---|---|---|---|---|---|---|
1 | He et al. [20] | ResNet-152, MobileNetV3 | 0.66 | 0.68 | - | 0.65 |
2 | Gjestang et al. [30] | Teacher-student framework | 0.89 | 0.89 | 0.89 | - |
3 | Barbhuiya et al. [22] | DarkNet | 0.71 | 0.71 | - | 0.74 |
4 | Galdran et al. [2] | CNN—BiT | 0.92 | - | - | - |
5 | Galdran et al. [21] | MobileNetV2, ResNeXt-50 | 0.64 | - | 0.65 | - |
6 | Proposed work | Novel feature fusion network | 0.97 | 0.97 | 0.98 | 0.98 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ramamurthy, K.; George, T.T.; Shah, Y.; Sasidhar, P. A Novel Multi-Feature Fusion Method for Classification of Gastrointestinal Diseases Using Endoscopy Images. Diagnostics 2022, 12, 2316. https://doi.org/10.3390/diagnostics12102316
Ramamurthy K, George TT, Shah Y, Sasidhar P. A Novel Multi-Feature Fusion Method for Classification of Gastrointestinal Diseases Using Endoscopy Images. Diagnostics. 2022; 12(10):2316. https://doi.org/10.3390/diagnostics12102316
Chicago/Turabian StyleRamamurthy, Karthik, Timothy Thomas George, Yash Shah, and Parasa Sasidhar. 2022. "A Novel Multi-Feature Fusion Method for Classification of Gastrointestinal Diseases Using Endoscopy Images" Diagnostics 12, no. 10: 2316. https://doi.org/10.3390/diagnostics12102316