Pest Identification and Control using Deep

Learning and Augmented Reality

Ascharya Soni Anuraag Khare
Department of Computer Science and
Darshan Balaji P S
Department of Computer Science and
Engineering, Department of Computer Science
Engineering,
Dr. Ambedkar Institute of Technology and Engineering,
2022 International Conference on Edge Computing and Applications (ICECAA) | 978-1-6654-8232-5/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICECAA55415.2022.9936053

Dr. Ambedkar Institute of Technology

Bengaluru - 560056, Karnataka, India Dr. Ambedkar Institute of Technology
Bengaluru - 560056, Karnataka, India
[email protected] Bengaluru - 560056, Karnataka, India
[email protected]
[email protected]
Sachin Verma Asha Rani K P
Department of Computer Science and
Gowrishankar S
Department of Computer Science and
Engineering, Department of Computer Science and
Engineering,
Dr. Ambedkar Institute of Technology Engineering
Dr. Ambedkar Institute of Technology
Bengaluru - 560056, Karnataka, India Dr. Ambedkar Institute of Technology
Bengaluru - 560056, Karnataka, India
[email protected] Bengaluru - 560056, Karnataka, India
[email protected]
[email protected]

Abstract — It is crucial to comprehend how insect pest populations

Augmented reality applications are developed in specialized 3D
affect the subsequent yield or harvest since the ultimate goal of
agriculture is to provide a sustained economic production of crop software to link animation or contextual digital information in
products. Using pesticides is the simplest technique to manage the the computer programme to an augmented reality "marker" in
pest infestation. However, using pesticides improperly or in excess the real environment. An augmented reality (AR) software or
can harm both people and animals as well as the plants. Machine browser plug-in on a computer device runs the marker's code
learning algorithms and image processing techniques are widely and layer the necessary picture or images as soon as it gets
used in agricultural research, and they have significant potential, digital data from a recognized marker.
particularly in plant protection, which ultimately leads to crop Using 3D assets or projected data insights enhances the physical
management. This paper highlights the detection of pests and their
world, which means the user's engagement with the corporeal
control measures. A smartphone camera will capture photographs
environment is not hindered but improved through an extension of
of the pests through a mobile app built using the Flutter
framework. The images are then analyzed in the app using various reality. So augmented reality is taking on a specific function. In
transfer learning based models for available pest identification this study, we highlight various machine learning methods for pest
kaggle dataset. The flutter framework offers the ability to monitor detection, with the findings shown in an Augmented Reality
targets in real-time on a mobile device and aids in visualizing the format. We give a literature review in the second section. The third
detected pest by integrating augmented reality on to the app. section compares the deep learning models for categorizing pests
that are based on transfer learning. In the fourth section, a sample
Keywords—Pest classification, Insect Detection, Augmented
mobile application for pest identification is shown. Finally, we
Reality, Image Processing, Machine Learning, Transfer Learning,
conclude our paper.
Deep Learning.

I. INTRODUCTION II. LITERATURE SURVEY

One of the biggest reasons for losses in agriculture is pests. Using various Machine Learning, Deep Learning, and Image
There are several chemical and biological pest management Processing model methods to precisely identify pests is covered in
techniques, but it is advised to take strict monitoring and control this section's literature review. This literature review's main aim is
procedures to ensure they are used to the fullest extent possible. to outline the advantages and potential uses of the various
The younger generations in India rarely receive formal training algorithms and pest identification methods.
in pest detection or pest control techniques. All pests are treated
Thenmozhi Kasinathan et al. [1], Wang and Xie dataset
the same way, with no consideration because some of them may
were experimented for the classification of nine and 24 insect
be beneficial and do not affect the plants, while others may also
classes respectively using machine learning algorithms such
feed on bugs that destroy crops. To solve this issue, we created
Artificial Neural Network (ANN), Support Vector Machines
a smartphone application for identifying pests that also makes
(SVM), K-Nearest Neighbor (KNN), and Naive bayes. The results
suitable control recommendations to satisfy the demands of
shown that the CNN model, using datasets from Wang and Xie,
farmers.
dataset, has the maximum classification accuracy of 91.5 percent
Depending on the pest it has identified, the application will and 90 percent for 9 and 24 classes of insects.
recommend management actions. The outcome will be shown
as augmented reality. Augmented reality is described as the Bo Wang et al. [2] have proposed a deep learning-based model for
simultaneous integration of digital information with the user's identifying crop illnesses and insect pests. Combining image
environment. A fresh layer of information is added to the processing and machine learning technologies improves the
existing world in augmented reality, as opposed to virtual performance of disease and insect pest recognition. The typical
reality, which produces a completely fabricated environment accuracy rate for identification is 96.26 percent.

Authorized licensed use limited to: DRDO-CABS. Downloaded on November 14,2022 at 04:29:41 UTC from IEEE Xplore. Restrictions apply.
Murray B Isman et al. [3], the issues of pest management in the currently included in identification systems, but only
21st century are compiled in this publication, along with the armyworms have real-time forecasting capabilities.
technology that may be employed to address them. The effects
of rising temperatures on the dynamics of pest populations have Cedric et al. [14], a pest detection application based on Cognitive
led to projections of heavy losses of yield of major grain crops computing and machine learning techniques which identifies crop
grown throughout the world, such as maize, wheat, etc. diseases and pests based on the image captured. But there are
certain API usage limits associated with Microsoft API providers.
Jayme Garcia Arnal Barbedo et al. [4] presents an article on Also, the result only displays the pest category and tags associated
Classifying and Detecting Pests Machine Learning and Using with it.
Proximal Images. They came to the conclusion that deep
learning methods particularly has been found to be
III. TRANSFER LEARNING-BASED DEEP
demonstrating a remarkable capacity to accurately detect as
LEARNING MODELS FOR PEST CATEGORIZATION
well as classify pests, whether in traps or natural photos.

Shrikant Salve et al. [5] makes use of augmented reality Pre-trained deep CNN models are used in a transfer learning-based
technology and a mobile camera to examine the crop disease. techniques for identifying pests. These models receive our dataset
The crop leaf image that was captured by the camera matches [15]. For categorization of images in agricultural pest recognition,
the online crop disease database. For crop disease prediction, which needs a sizable dataset coupled with powerful computing,
they suggested a mobile-based augmented reality application. deep learning methods like CNN have proved successful. Large
They have successfully implemented the prototype of a mobile datasets for the agriculture area are typically difficult to get
app that uses augmented reality and can find disease on crop because of regional weather, insect indiscriminacy, and abandoned
leaves. A photo of the crop leaf that was harmed has been used environments for chronic pests. In order to categorize data,
as a marker. researchers employed a transfer learning model that draws on
training data from a huge dataset. The CNNs include a variety of
Mohanty et al. [6], AlexNet [7] and GooLeNet [8] deep learning filters and kernels made of trainable parameters that may spatially
models were used to identify plant diseases using images, convolve on an input picture to recognise characteristics like edges
these two deep learning models have been compared. They and shapes. For first stage we do resizing of 224,224. For second
conducted their experiments using a collection of 54,306 stage, networks like EfficinetNetB7 or Xception is used. For third
images of good as well as damaged leaf images from 14 stage GlobalAveragePooling2D layer is used which is followd by
varieties of crops and 26 types of diseases. The outcome flatten and dense layer with shape 128 and relu activation function.
demonstrates GoogleNet outperforms AlexNet in their research Sixth stage is batch Normalization which is again followed by
job. dense layer with shape 512 amd relu activation. The last layer is
dense layer with shape of 9 and softmax activation function. At
In the study [9], they adjusted a few of the hyperparameters in last, Adam optimizer is used with learning_rate of 1e-5, beta_1 of
the deep learning-based models Cifar10 [11] and GooLeNet in 0.9 and beta_2 of 0.99.
order to increase accuracy. 500 photos of maize leaves are
included in the dataset. When the training-testing ratio is 80-20, Dataset from the kaggle notebook of Simran Volunesia [15] is
for the GoogLeNet model they have reached average accuracy used. According to the model's requirements, we adjusted the pest
levels of 98.9 .For the Cifar10 model, the accuracy reached photos into the desired shape, the pest dataset was then separated
was 98.8 percent. They created CaffeNet, a CNN-based deep into 70% for training and the rest 30% for test. Each model was
learning model that can identify 13 different types of diseases trained over four trials using 10 epochs with a batch size of 16 and
and 2589 pictures, using the Caffe [10] framework. They have a learning rate of 0.01. Python programming language was used
processed the database's images through picture resizing and for all experiments, together with Keras and TensorFlow. Four
enhancement. The 10-fold cross validation approach is used to different types of CNN models were investigated: Xception,
measure accuracy. Using finetuning, it can attain an accuracy MobileNet, MobilenetV2 and EfficientNetB7. The ImageNet
of 96.3 percent on average. Large Scale Visual Recognition Challenge saw these networks
perform quite well (ILSVRC).
Wikilimo et al. [12]—Wikilimo’s is a Lightweight Deep
Learning Model for Automated Pest Detection. It is an app that The feature extraction results show that simple backgrounds have
analyzes pest images to determine its type and the crops it a classification accuracy that is 5.9 percent greater than that
affects. The developers employed deep learning models, which of complicated backgrounds. The accuracy of four distinct transfer
were pre-trained on the IPV102 database and fine-tuned to learning models is contrasted in the accompanying graph. Even
operate in this scenario. The model had low accuracy and could though EfficientNetB7 is significantly more accurate than the
not specify which crops the pest would affect. other three models, all four have comparable accuracy
percentages.
Pest and Crop Management application by UofM Mobile[13] is
a mobile application for identifying pests, insects and weeds.
The app will help users adopt integrated pest control for
Canadian field crops.Insect pests of canola and wheat are

Authorized licensed use limited to: DRDO-CABS. Downloaded on November 14,2022 at 04:29:41 UTC from IEEE Xplore. Restrictions apply.
 Table II. DETAILS OF AVAILABLE PEST IDENTIFICATION APPS

APPLICATION DEVELOPER STATUS

Wikilimo [20] Wikilimo Inactive

Pest and Crop Management UofM Mobiles Active

Cedric[21] Raj Gopal Active

Pest Identification-Kisaan is a mobile application that was

Fig. 1. Graph on accuracy comparison for different Transfer Learning developed using flutter framework and machine learning. The Fig.
Model 3.1-3.8 screenshot shows various features available in the app such
as login functionality to login the app, home page will be there for
news and recommendation, Menu bar for different options , multi-
Table I : DETAILS OF PRE-TRAINED MODELS.
language support across the app, weather forecasting of next 5
days and expert chat support window along with pest detection
Deep Input Convo- Pooling Non- Train- scanner page. The images captured/streamed through the app
CNN Shape lution Layers trainable able camera interface are passed onto transfer learning model
Model Layers Parameter Param integrated in .tflite format to flutter codebase to give pest
eter
classification output. As Streaming of pest/image is live through
Xception (224,224,3) 37 4 2,08,65,480 20,490 camera it continuously match the image/pest with the integrated
model , code and check the matching percentage and as it reaches
MobileNet (224,224,3) 14 1 42,53,864 10,010 threshold equal or above 95 % insect is identified and detailes
related to it is called through API. Fig.3.8 – Fig.3.9 show the result
EfficientNetB7 (224,224,3) 51 1 2,35,67,800 20,490 of different pests like MITES,BEETLE,STEM-BORER and their
pest description, measure control and affected crops are then
MobileNetV2 (224,224,3) 34 1 22,57,984 12,810 displayed in Augmented Reality format.

We can see from the above table the comparison between

different pre-trained deep learning models. It describes the
various shapes of input data that are fed to different models and
the trainable and non-trainable data present along with other
metadata.

Review of this study reveals that EfficientNetB7 technique had

a classification accuracy of 95.51 percent for the dataset [15].
Xception had a classification accuracy of 94.3 percent, and
MobileNet had a classification accuracy of 94.5 percent. When
compared to other approaches using our dataset [15], the
suggested method has greater accuracy rates.

IV. PEST IDENTIFICATION MOBILE APPLICATION

This section presents comparision of previously available

mobile applications for identifying pests using image
processing methods. Images of the pests are uploaded to the
app, which are then examined. The accuracy percentage is Fig. 2 App Workflow
calculated, and the type of pest is determined. Our project app
detects pests and provides their control measures as Augmented
Reality and has better accuracy compared to existing apps.

Authorized licensed use limited to: DRDO-CABS. Downloaded on November 14,2022 at 04:29:41 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3.1 Login Fig. 3.2 English version Fig. 3.3 Hindi version

Fig. 3.4 Kannada Version Fig. 3.5 Expert View Fig. 3.6 Chat Page

Authorized licensed use limited to: DRDO-CABS. Downloaded on November 14,2022 at 04:29:41 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3.7 Multi Languagge Weather Forecase Pages

Fig. 3.8. Mites and Beetle Pests Detection through Augmented Reality.
Authorized licensed use limited to: DRDO-CABS. Downloaded on November 14,2022 at 04:29:41 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3.9. Stem borer Pest Detection through Augmented Reality.

1211

Authorized licensed use limited to: DRDO-CABS. Downloaded on November 14,2022 at 04:29:41 UTC from IEEE Xplore. Restrictions apply.
 IV. CONCLUSION “Identification of plant diseases using convolutional neural
networks”, International Journal of Information Technology. .
The evaluation based on the aforementioned three Int. j. inf. tecnol. 13, 2461–2470, 2021
approaches, namely EfficientNetB7, Xception, Mobilenet [10]. Jia, Yangqing, Evan Shelhamer, Jeff Donahue, Sergey Karayev,
and MobileNetV2, trained using Kaggle dataset [15]. It is Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor
found that EfficientNetB7 method has a classification Darrell, "Caffe: Convolutional architecture for fast feature
accuracy of 95.15% for the dataset [15]. When you lack a embedding," In Proceedings of the 22nd ACM international
huge dataset and powerful computing capabilities, you can conference on Multimedia, pp. 675-678. 2014.
employ a transfer-learning-based algorithms. Based on the [11]. Marko Arsenovic, Andras Anderla, Sladojevic, Srdjan, Dubravko
Culibrk, and Darko Stefanovic, "Deep neural networks based
mobile apps survey done for pest identification as depicted in
recognition of plant diseases by leaf image classification,"
table II, currently available apps are inactive, this gives us a
Computational intelligence and neuroscience , Vol. 2016, doi:
need for developing an efficient functional app. 10.1155/2016/3289801, June 2016
The model trained using EfficientNetB7 was then linked to [12]. Wikilimo app, “https://github.com/wikilimo/Wikilimo”, 2019.
the newly created mobile application called Kisaan. It was [13]. UofM Mobiles, “https://appgrooves.com/app/pest-and-crop-
management-by-university-of-manitoba”,
developed using a flutter framework. The images captured
[14]. RajGopal, “https://github.com/rajagopal28/Cedric”,2016.
through the app are passed onto the transfer learning model
[15]. “https://www.kaggle.com/datasets/simranvolunesia/pest-dataset”
to give pest classification output. The outcome of the pest [16]. Jadhav, S.B., Udupi, V.R. & Patil, S.B. Identification of plant
detection is represented by scanning a pest image and the diseases using convolutional neural networks. Int. j. inf. tecnol. 13,
output is then presented in augmented reality form. 2461–2470 (2021). https://doi.org/10.1007/s41870-020-00437-5

In the promising future of pest control, when many

technologies will communicate with one another and require
less human interaction, future scope of our app will include
the feature of - Crop disease detection (what happened to
plant and how) adding additional pests in dataset, Soil
Quality detection (what's the temperature and quality to grow
some crop). Also, adding more languages for pan- India.
Optimizing Augmented Reality for better user experience to
detect different things easily and swiftly.

REFERENCES

[1]. Thenmozhi Kasinathan, Dakshayani Singaraju, Srinivasulu

Reddy Uyyala,”Insect classification and detection in field
crops using modern machine learning techniques”, September
2020
[2]. Bo Wang, “Identification of Crop Diseases and Insect Pests
Based on Deep Learning”, Volume 2022 Article ID 9179998
[3]. Murray B Isman, “Challenges of Pest Management in the
Twenty First Century”, doi: 10.3389/fagro.2019.00002
[4]. Jayme Garcia Arnal Barbedo, “Detecting and Classifying Pests
in Crops Using Proximal Images and Machine Learning: A
Review”, Embrapa Agricultural Informatics, doi:
10.3390/ai1020021, June 2020
[5]. Shrikant Salve, “Augmented reality app for pest detections” in
17th IFIP TC.13 International Conference on Human-
Computer Interaction, September 2020
[6]. Mohanty, Sharada P., David P. Hughes, and Marcel Salathé,
"Using deep learning for image-based plant disease detection."
Frontiers in plant science 7, doi:10.3389/fpls.2016.01419,
September 2016.
[7]. Kala H.S.; Ramachandra Hebbar; Anjali Singh S.; Amrutha R.
Amith R. Patil; Deepak Kamble, “AgRobots (A Combination
of Image Processing and Data Analytics for Precision Pesticide
Use)”, in Proceedings of IEEE Graphic and image processing
in general, Vol. 2018, Issue ICDI3C New York, USA, 16
August 2018
[8]. Vishakha B. Sanghavi, Dr. H. B. Bhadka, Vijay J. Dubey .“A
Review of Various Convolutional Neural Network Model for
Crop Pest Recognition and Classification” IJERT 2278-0181
[9]. Sachin B. Jadhav, Vishwanath R. Udupi, Sanjay B. Patil,

Authorized licensed use limited to: DRDO-CABS. Downloaded on November 14,2022 at 04:29:41 UTC from IEEE Xplore. Restrictions apply.