2024 3rd IEEE International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024)

Medical Plant Detection using Transfer learning

Coupled with K-Nearest Neighbour Algorithm
2024 3rd International Conference on Artificial Intelligence For Internet of Things (AIIoT) | 979-8-3503-7212-0/24/$31.00 ©2024 IEEE | DOI: 10.1109/AIIoT58432.2024.10574671

Sai Subramanian Srambickal Aman Srivastava Somasundaram S. K.

Department of Computer Science and Department of Computer Science and Department of Information Security
Engineering Engineering School of Computer Science and Engineering,
School of Computer Science and Engineering, School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamil
Vellore Institute of Technology Vellore, Tamil Vellore Institute of Technology, Vellore, Tamil Nadu, India
Nadu, India Nadu, India [email protected]
[email protected] [email protected]

Abstract—Medical Plants are being used for more than a

thousand year, with evidences dating before the Mauryain Era II. RELATED WORKS
(around 322 BCE), they are widely used in the Ayurvedic Plant and property Identification is extremely important when
school of medicine for therapeutic as well as medicinal it comes to the use case of medicine, since this is a very
purposes, it requires an expert to identify the species as well as sensitive field which deals with human health, misidentification
the medical property associated with a given plant. This paper cannot be afforded. Recent Advances in Deep learning have
aims to use deep learning in order to identify a given plant helped us streamlining this task in terms of identification time as
using the leaves of the plant by using a pretrained well as giving accurate predictions even when there are no
Convolutional Neural Network (CNN) model namely experts present on site, which is very helpful for rural and remote
ResNet50 and MobileNet to extract the latent vector which will areas where experts are not easily available and due to this
then be passed to K-Nearest Neighbour (KNN) algorithm in various research have been conducted in the field.
order to identify the plant class, the best accuracy of the [1] This research explores a new image classification model
proposed models comes out to be 97.65%. achieving an accuracy of 97.43%. The authors developed a
Index Terms—Deep learning, Plant identification, custom version of MobileNet V3, a known efficient model
MobileNetv3, ResNet50, K-nearest neighbour architecture. Uniquely, they incorporate three global average
pooling layers, each summarizing the image at different
I. INTRODUCTION dimensions. These summaries are then combined to create a
comprehensive feature vector. This feature vector is fed into a
The use of Plants for creation of therapeutic and medicinal
final layer with a Softmax function to predict the most likely
compounds predates human history. In India, for thousands of
class out of 11 possible categories. This approach leverages the
years, medicinal plants haven’t just been treatments, they’ve
efficiency of MobileNet V3 while potentially improving
been woven into the very fabric of the culture. Ancient medical
classification through the novel use of triple pooling.
systems like Ayurveda documented the use of countless plants
One of the most inspiring academic literatures for this study
for various ailments, while the holy basil plant graces homes and
comes from the paper [2], this paper studies the architecture
festivals, revered for its healing properties. Traditional healers
where Modified VGG16 architecture were used to architecture
possess an incredible knowledge passed down through
which will be able to outperform the existing calculate the feature
generations to accurately identify these plants, a crucial step in
vector of the images from and gradient boosting which were then
harnessing their benefits and avoiding harm. This rich tradition
used to classify the Vietnamese herbal plants.
continues to this day, with medicinal plants remaining a
[3] This research paper explores a method for image
significant source of healthcare, especially in rural areas.
classification using deep learning. The approach utilizes
Modern science joins hands with these age-old practices, using
MobileNet V2, a powerful but efficient pre-trained model, to
sophisticated methods alongside traditional knowledge to
analyze images. MobileNet V2 doesn’t directly predict the image
identify plants and unlock their full medicinal potential, ensuring
class. Instead, it acts like a magnifying glass, identifying key
this herbal legacy thrives in the future.
characteristics within the image. These characteristics, called
The main objective of this study is to present with a new
feature biomarkers, are then used by the model to predict the
architecture which will be able to outperform the existing
most likely category for the image. In essence, the paper
Medical Plant image classification problem. This study proposes
combines the feature extraction power of MobileNet V2 with a
a coupled approach to predict the class of the images using the
separate model to categorize the images based on the extracted
latest state of the art models namely ResNet50 and MobileNet-
information.
V3 which are used to extract the features which will then be
[4] This research appears to focus on real-time object detection
given to KNN algorithm in order to predict the class of the plant.
on hardware rather than developing a complex model from
The subsequent sections are discussed in the paper, Literature scratch. The authors likely employed a basic convolutional
survey, Methodology, Results and Discussion, Conclusion and neural network (CNN) architecture. Instead of spending
References. significant effort on creating a highly sophisticated model, they
concentrated on extracting features called HOG (Histogram of
979-8-3503-7212-0/24/$31.00
Authorized ©2024
licensed use limited to: FLORIDA IEEE UNIVERSITY. Downloaded on August 20,2024 at 16:46:30 UTC from IEEE Xplore. Restrictions apply.
INTERNATIONAL
2024 3rd IEEE International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024)

Oriented Gradients) from the images. HOG features capture the development of a mobile application for easy access to plant
edge and gradient information in an image, which can be information through QR code scanning in herbal gardens.
helpful for object detection. Notably, the emphasis seems to [8] In this study, classification of medicinal plants has been
be on optimizing the entire system for real-time object done using self-created dataset of 1500 images of locally
detection on hardware. This suggests the authors might have available common medicinal plant leaves. The classification of
explored techniques to streamline the model and feature medicinal plants is conducted by employing 4 stage process i.e.
extraction for efficient execution on specific hardware platforms, (i). Pre-processing (ii). Segmentation (iii). Feature Extraction
potentially for embedded systems or mobile devices. (iv). Training and testing using the classifier. The study discusses
[5] This paper presents a novel method for classifying and the importance of identifying and classifying medicinal plants
recognizing 18 types of medicinal plants using a ResNet34 using machine learning and deep learning technologies. The
convolutional neural network”. The method leverages attention experiment focuses on classifying locally available medicinal
maps, CutMix data augmentation, and a class-balanced label plants using leaf images, achieving an ac- curacy of 98.7%. It
smoothing loss function to enhance accuracy. The experiment also includes a literature review of various algorithms and
demonstrates an accuracy rate of 96.8% on a grassland plant techniques used for plant image classification, highlighting the
dataset. The ResNet34, Attention-Net, and Plant-Net models are importance of feature extraction and its impact on classifier
compared, with Plant-Net achieving the highest accuracy. The performance. Additionally, it discusses the use of convolutional
dataset consists of 18 plant species from the China Plant Image neural networks (CNN) and the experimental results, concluding
Database. The study focuses on improving plant species with the need for continued research in classifying medicinal
identification through deep learning and attention- based feature plants, especially by incorporating both leaves and flowers and
extraction. collecting more plant species for classification.
[6] This study discusses the development of an auto- mated [9] The research paper presents the development of the
classification system for identifying Ayurvedic medicinal plants Domain Specific Modelling Language System (DOMLS), a
using machine learning and computer vision techniques”. It system designed to facilitate the search for medicinal plants
emphasizes the need for accurate and convenient plant using a deep ontology and deep learning techniques. The system
identification due to the global reliance on medicinal herbs allows users to search for medicinal plants based on various
and the growing commercialization of Ayurvedic medicine. The criteria, including names, functions, diseases, taboos, and
proposed system involves a four-step process: image capture, valuable parts. The system utilizes a deep learning model,
preprocessing, feature extraction, and classification, utilizing YOLOv5, for image classification and recognition, achieving
supervised machine learning principles. The authors also high accuracy when tested with a dataset of 6000 images of 30
highlight the significance of image processing algorithms, such species of medicinal plants. The dataset is annotated and divided
as average filtering and RGB to grayscale conversion, in into two sets: 5400 images for training the model and 600
recognizing and obtaining leaf properties for machine learning images for evaluating the model’s accuracy. The experimental
approaches. The suggested approach has been evaluated using results of the YOLOv5-based engine demonstrate high accuracy,
669 samples of Ayurvedic leaf pictures from 10 distinct species, with a mean average precision (mAP) of 99.3% and a recall rate
achieving an accuracy of 98.0597% in identifying medicinal of 99.6%. The system’s ontology-based search engine was also
plants based on leaf images. The study also references various evaluated, showing promising results in expanding search results
studies and methodologies for automated plant leaf analysis, with ontology, providing users with advanced information based
showcasing the potential of machine learning algorithms in on their search queries. The system’s ability to identify medicinal
improving the performance of Ayurvedic medication. Overall, plants through images and keywords was tested, and the results
the research aims to address the challenges in identifying showcased the capability of the system to accurately recognize
medicinal plants and emphasizes the potential of machine and provide information on medicinal plants based on user input.
learning and computer vision in this domain. Because of diminishing gradients, regular neural networks for
[7] This study discusses the development of an automated image identification have difficulty with depth. ResNets by [10],
system for identifying Ayurvedic medicinal plant leaves using solve this by introducing shortcut (Fig.1) links that enable data
deep learning, particularly convolutional neural networks to pass through the network without alteration. Bypassing the
(CNN)”. The research aims to address the need for accurate vanishing gradient issue, these connections teach the network to
identification of Ayurvedic plants due to their significance in distinguish between the original input and the altered output.
treating various diseases with fewer side effects compared to ResNets may now be far deeper, capturing more intricate features
conventional medicines. The authors created a dataset of 4390 and improving picture identification precision as a result.
images of 35 different species of medicinal leaves, which was
used to train and test the CNN-based system. Various image
processing techniques, such as data augmentation and pre-
processing, were employed to enhance the accuracy of the
model, resulting in a reported accuracy of 94.10%. The study
also involved collaboration with Ayurveda doctors and experts
to ensure the authenticity and reliability of the dataset and the
model’s performance. The proposed CNN model automatically
extracts features from images, making it suitable for plant
classification The potential future applications of the developed
system include the integration of additional features to provide Fig. 1. Residual Block, as mention in [10].
information on Ayurvedic plants’ medicinal properties and the
Authorized licensed use limited to: FLORIDA INTERNATIONAL UNIVERSITY. Downloaded on August 20,2024 at 16:46:30 UTC from IEEE Xplore. Restrictions apply.
2024 3rd IEEE International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024)

Convolutional neural networks (CNNs) of the MobileNet by III. METHODOLOGY

[11] kind are made especially for mobile and embedded
This section outlines the computational environment and
applications. Because MobileNet is efficient and lightweight
model parameters employed in this study.
compared to larger CNNs, it is the perfect choice for situations
For the architecture specifications we have used two pre-
where processing power and battery life are critical. Depth wise
trained models namely ResNet50 by [10] and MobileNetV3 by
separable convolutions are used by MobileNet. These reduce the
[11] both are pretrained in order to utilize the power of
number of computations required for ordinary convolutions by a
transfer learning which was inspired by [1], these state-of-the-art
significant amount without compromising accuracy. As a result,
models are used to extract features and create a latent vector
MobileNet can operate far faster and with less memory than
which will then be passed to KNN algorithm which is used to
conventional CNNs.
predict the class of the plant.
The literature review that is currently available highlights
In this research, we implemented deep learning models for the
significant developments in machine learning (ML) techniques,
classification of Indian medicinal plant leaves using the PyTorch
especially in the field of computer vision. Still, there is a
framework. PyTorch was chosen due to its flexibility and ease of
significant lack of research on the combination of pre- trained
use in data loading tasks compared to other deep learning
Convolutional Neural Network (CNN) architectures and
frameworks.
lightweight machine learning techniques like K-Nearest
Neighbors (KNN). Although earlier research has shown CNNs to
Data Preparation:
be effective, nothing is known about the possible benefits of
• We utilized a dataset from Kaggle containing images of 80
combining them with more straightforward but understandable
distinct Indian leaf varieties known for their potent
algorithms such as KNN. This work attempts to bridge this gap by
medicinal properties. The dataset comprised a total of 5,945
examining the combination of pre-trained CNNs and KNN, with
images, Dataset sample Fig:2 and Fig.3:.
the goal of utilizing the advantages of both techniques to improve
• For model training and validation, the data was split into
performance, especially in scenarios with limited resources and
two sets:
interpretability-demanding jobs. This work attempts to clarify
the advantages of this integrated strategy through empirical – Training set: 4,576 images (77%)
evaluation, adding to a better comprehension of the – Validation set: 1,189 images (23%)
complementary relationship between lightweight ML algorithms
and sophisticated neural network architectures. Model Architecture and Training:
1) Deep Feature Extraction:
• We employed pre-trained convolution neural net-
works (CNNs) as feature extractors. These models
were pre-trained on the large-scale ImageNet dataset,
allowing them to capture generic image features. In
our research, we explored two pre-trained CNN
architectures the reason for selecting these two CNN
architecture specifically was because both of these are
very well known architectures, ResNet was
specifically chosen because we want to reduce the
vanishing gradient issue so that when the latent vector
is passed to the KNN for classification it gives
improved accuracy Fig. 4, whereas MobileNet was
chosen because since it is computationally efficient
Fig. 2. Alovera Sample from dataset
architecture it can be used this model in fields where
it can even on low computational devices such as
embedded systems :
– ResNet: A deep residual learning architecture
known for its effectiveness in image recognition
tasks.
– MobileNet: A lightweight CNN architecture de-
signed for efficient computation on mobile and
embedded devices.
• Both models were fine-tuned on our plant image
dataset. During fine-tuning, only the final layers of
the pre-trained models were updated, while the earlier
layers (responsible for capturing generic features)
were kept frozen. This approach leverages the pre-
trained knowledge while adapting the models to the
Fig. 3. Ashwagandha Sample from dataset
specific task of plant leaf classification.
• The pre-trained models extract a fixed-length latent
Authorized licensed use limited to: FLORIDA INTERNATIONAL UNIVERSITY. Downloaded on August 20,2024 at 16:46:30 UTC from IEEE Xplore. Restrictions apply.
2024 3rd IEEE International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024)

vector representation for each input image. This la- • J(θt; xi, yi) is the loss function with respect to the parameters
tent vector encapsulates the essential image features θt given the input data xi and corresponding target yi.
learned by the CNN. • ∇ denotes the gradient operator, and ∇J(θt; xi, yi) is the
• gradient of the loss function with respect to the
2) K-Nearest Neighbors (KNN) Classification: The latent
parameters θt evaluated at the data point (xi, yi).
vectors extracted from the pre-trained CNNs were fed into
a K-Nearest Neighbors (KNN) algorithm for final class Evaluation:
prediction. KNN classifies data points based on their The performance of the proposed models was evaluated on
similarity to the K nearest neighbors in the training data. In the held-out validation set. We employed standard classification
our case, the KNN classifier utilized the latent vectors to metrics like accuracy, precision, recall, and F1-score to assess
determine the class (medicinal plant variety) of each the model’s ability to correctly classify the medicinal plant
unseen image. varieties.

3) Loss Function and Optimization: IV. RESULTS AND DISCUSSION

• We employed the cross-entropy loss function to
measure the difference between the predicted class The section discusses the results that were generated using the
probabilities and the true labels during training. The methods that were discussed in the previous section. Tabel 1
cross-entropy loss is a common choice for multi- class shows that then model architecture that we discussed is at par
classification tasks. with the other architectures specifically that were discussed in
the paper [2]. Proposed model ResNet with random forest
• Stochastic Gradient Descent (SGD) was used as the
97.65% accuracy give the higher validation accuracy compared
optimization algorithm to update the weights and
biases of the KNN model based on the calculated loss. to the ResNet with KNN architecture 97.39% accuracy, whereas
the highest validation accuracy is given by ResNet50 with
SGD iteratively adjusts the model parameters in the
direction that minimizes the loss function. PyTorch’s Random Forest, so as we can infer that the MobileNet version
with the KNN outperforms the random forest MobileNet.
built-in functionalities were leveraged to efficiently
implement the loss calculation and backpropagation The proposed MobileNet with KNN 97.31% accuracy out-
for gradient computation. performs the MobileNet with random forest 96.89% accuracy
and this is due to the fact that KNN is a much simpler algorithm
than Random Forest and thus reduces complexity and Biases
while computation due to which the model with KNN (proposed
model) was able to perform better

TABLE I. ACCURACY COMPARISON

Coupling ResNet50 MobilNet

KNN 97.39% 97.31%
RandomForest 97.65% 96.89%

we can also infer from the loss curves that the model is
converging and the proposed model is at par with the current
models.

Fig. 4. Architecture Diagram of proposed model

The stochastic gradient descend uses learning rate, features

and labels both predicted and true labels to update the trainable
parameters of the model, its formula is as follows:
Fig. 5. ResNet-KNN loss curve.

θt+1 = θt − ηt∇J(θt; xi, yi) (1) In particular, the MobileNet design with KNN outperforms the
where ResNet50 architecture in terms of validation accuracy,
demonstrating how well it handles the categorization task.
• θt represents the parameters at time t.
Furthermore, even though ResNet50 in conjunction with Ran-
• ηt represents the learning rate at time t.
dom Forest yields the best validation accuracy, it is important to
Authorized licensed use limited to: FLORIDA INTERNATIONAL UNIVERSITY. Downloaded on August 20,2024 at 16:46:30 UTC from IEEE Xplore. Restrictions apply.
2024 3rd IEEE International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024)

note that the MobileNet model that incorporates KNN performs for the classification of medicinal plants using Transfer learning
better when put side by side with the MobileNet model that uses from the ImageNet dataset on which the ResNet50 and
Random Forest. This implies that the MobileNet architecture’s MobileNet was pretrained and further it was fine-tuned with the
classification power is greatly influenced by the algorithm medicinal plant dataset.
selected for integration. Computational Resources:
We leveraged Google Kaggle’s cloud environment for model
training and experimentation. This environment provided access
to the following hardware resources:
• RAM: 29 GB
• GPUs: 2x Nvidia T4 GPUs with 15 GB vRAM each
• CPU: Intel Xeon 2.20 GHz CPU with 4 cores
These resources ensured sufficient computational power for
training deep learning models on our plant image dataset.

V. CONCLUSION
In conclusion, this work presented a novel architecture for
plant image classification on mobile and embedded devices. The
architecture leverages the efficient feature extraction capabilities
Fig. 6. MobileNet + KNN loss curve
of MobileNet and the classification strengths of K- Nearest
Neighbors (KNN). Empirical evidence demonstrates that this
combination achieves superior accuracy (97.31%) compared to
existing approaches. This finding suggests the potential for
deploying powerful and resource-constrained plant image
classifiers on mobile phones and embedded systems.
Future research should focus on optimizing the model for even
greater accuracy and efficiency while maintaining mobile
compatibility. Additionally, testing on a wider variety of plant
datasets and exploring techniques to handle imbalanced data are
crucial for generalizability. Furthermore, the paper identifies an
interesting avenue for future exploration: automatically
clustering unknown plant images based on medicinal properties.
This functionality could hold significant value for applications in
healthcare and natural medicine.
Fig. 7. ResNet + Randomforest loss curve In essence, this work contributes to the development of mobile
and embedded plant image classification by proposing a novel
architecture with demonstrably high accuracy. By addressing the
limitations and exploring promising future directions,
researchers can build upon this foundation and unlock the full
potential of mobile plant identification tools, with potential
applications extending beyond agriculture into the realm of
mobile healthcare.

ACKNOWLEDGMENT
We would like to express our gratitude to the Kaggle
community for providing a valuable platform for data science
exploration and collaboration. This research would not have been
possible without the Indian Medicinal Plant Image Dataset by
Fig. 8. MobileNet + Randomforest loss curve Chirag Chauhan (kaggle Master), which served as the foundation
for our analysis. We acknowledge the effort of the dataset
Furthermore, the loss curves acquired throughout the training creators in making this data publicly available.
process offer additional understanding of each architecture’s
learning dynamics. The models’ convergence behavior and
generalization abilities are explained by these curves. In- depth REFERENCES
examination of these curves in conjunction with the attained [1] Daryl B. Valdez, Chris Jordan G. Aliac, and Larmie S. Feliscuzo.
Medicinal plant classification using convolutional neural network and
performance measures, allows for a thorough com- prehension transfer learning. In 2022 IEEE International Conference on Artificial
of the effectiveness and resilience of the suggested techniques. Intelligence in Engineering and Technology (IICAIET), pages 1–6, 2022.
The outcomes demonstrate how well the suggested MobileNet [2] Anh Vo and Bao Thien. Vietnamese herbal plant recognition using deep
convolutional features. International Journal of Machine Learning and
design with KNN integration performs in terms of attaining Computing, 9, 05 2019.
competitive results for the given classification problem. These [3] Jafar Abdollahi. Identification of medicinal plants in ardabil using deep
results further the development of machine learning techniques learning: Identification of medicinal plants using deep learning. In 2022
Authorized licensed use limited to: FLORIDA INTERNATIONAL UNIVERSITY. Downloaded on August 20,2024 at 16:46:30 UTC from IEEE Xplore. Restrictions apply.
2024 3rd IEEE International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024)

27th International Computer Conference, Computer Society of Iran

(CSICC), pages 1–6, 2022.
[4] Rayellee Myrtle Laire Ang and Noel B. Linsangan. Herbal medicinal plant
identification using leaf vein through image processing and con- volutional
neural network. In TENCON 2023 - 2023 IEEE Region 10 Conference
(TENCON), pages 1046–1051, 2023.
[5] Yang Liu, Yating Li, Yanjie Zhao, and Xitai Na. Image classification and
recognition of medicinal plants based on convolutional neural net- work.
In 2021 IEEE 21st International Conference on Communication
Technology (ICCT), pages 1128–1133, 2021.
[6] Subhashree Darshana and Kasturi Soumyakanta. A revolutionary machine-
learning based approach for identifying ayurvedic medicinal plants. In
2022 International Conference on Advancements in Smart, Secure and
Intelligent Computing (ASSIC), pages 1–5, 2022.
[7] Swati G. Kale, Shoaib Ansari, Tawfeek Khan, Mansi Chitriv, Harshit
Sathone, Hariyali Mansata, P.G. Jaiswal, N. R. Wankahde, and Roshan
Umate. Identification of ayurvedic leaves using deep learning. In 2023
International Conferenceon Communication, Circuits, and Systems (IC3S),
pages 1–6, 2023.
[8] Rajani S and Veena M.N. Ayurvedic plants identification based on
machine learning and deep learning technologies. In 2022 Fourth In-
ternational Conference on Emerging Research in Electronics, Computer
Science and Technology (ICERECT), pages 1–6, 2022.
[9] An C. Tran, Thien Khai Tran, Nguyen Minh Nhut, and Nguyen Huu Van
Long. Building a deep ontology-based herbal medicinal plant search
system. International Journal of Information Technology: An Official
Journal of Bharati Vidyapeeth’s Institute of Computer Applications and
Management, pages 1 – 15, 2023.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual
learning for image recognition, 2015.
[11] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko,
Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.
Mobilenets: Efficient convolutional neural networks for mobile vision
applications, 2017.

Authorized licensed use limited to: FLORIDA INTERNATIONAL UNIVERSITY. Downloaded on August 20,2024 at 16:46:30 UTC from IEEE Xplore. Restrictions apply.