Received March 21, 2022, accepted March 31, 2022, date of publication April 13, 2022, date of current

version April 22, 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3167061

Learning Aided System for Agriculture

Monitoring Designed Using Image
Processing and IoT-CNN
KANDARPA KUMAR SARMA 1 , (Senior Member, IEEE), KUNAL KINGKAR DAS 1 ,
VIKASH MISHRA1 , SAMADRITA BHUIYA1 , AND DMITRII KAPLUN 2,3 , (Member, IEEE)
1 Department of Electronics and Communication Engineering, Gauhati University, Guwahati, Assam 781014, India
2 Department of Automation and Control Processes, Saint Petersburg Electrotechnical University ‘‘LETI,’’ 197022 Saint Petersburg, Russia
3 Faculty of Information Technologies, Kazakh-British Technical University, Almaty 050000, Kazakhstan
Corresponding author: Dmitrii Kaplun ([email protected])
This work was supported by the Ministry of Science and Higher Education of the Russian Federation under Grant FSEE-2020-0002.

ABSTRACT The Internet of Things (IoT) and artificial intelligence (AI) based methods for monitoring,
control, and decision support are combined to design of a smart agriculture assistance system. The proposed
system has a sensor pack that provides continuous data capture of temperature records, air and soil moisture
and a camera for obtaining near-infrared (NIR) images of the plant leaves for use with an AI decision support
system. We identify twelve types of vegetation for the study, out of which five disease classes of the tomato
leaves are categorized using a Convolutional Neural Network (CNN). The work also includes experiments
conducted with multiple clustering-based segmentation methods and some features namely Gray level
co-occurrence matrix (GLCM), Local binary pattern (LBP), Local Binary Gray Level Co-occurrence Matrix
(LBGLCM), Gray Level Run Length Matrix (GLRLM), and Segmentation-based Fractal Texture Analysis
(SFTA). Out of several AI tools, CNN proves to be effective in providing automated decision support for
classifying the plant leaf disease types through a cloud server that can be accessed using an app. Extensive
on-field trials show that the system (VGG16 CNN, GLCM and a fuzzy based clustering) is effective in hot
and humid conditions and proves to be reliable in identifying disease classes of certain vegetable types,
certain usable vegetation cover of farmland and regulation of watering mechanism of crops.

INDEX TERMS Artificial intelligence, near-infrared images, CNN, image processing, leaf disease, smart
agriculture.

I. INTRODUCTION towards using data-aided tools, especially artificial intelli-

World over agriculture is one such human activity that criti- gence (AI) based methods that are effective in providing yield
cally ensures food supply to the ever-increasing human pop- fore-cast, process monitoring, accurate control, and reliable
ulation. This has necessitated the adoption of technology decision support [4]. The advantage of using AI tools like
and other modern means to enhance the productivity of the Artificial Neural Network (ANN), Deep Neural Network
agriculture sector to meet the increasing demand for food (DNN), etc., is the fact that these can learn from the surround-
required to sustain the expanding human population all over ings, retain the learning and use it subsequently [5].
the world [1]. With the proliferation of the internet and digital For a majority of the countries, including India, the use
technology, the agriculture sector has also adopted the best of pattern recognition and AI-aided tools in agriculture
practices of these developments to explore ways and means becomes more pertinent because a sizable section of the
to increase productivity, ensure efficiency, and contribute to productive workforce is engaged in this sector. Subsequently,
protecting the environment [2], [3]. Lately, the trend has been the mechanism embraces precision approaches increasing
efficiency and productivity in agriculture. Over the years,
The associate editor coordinating the review of this manuscript and attempts have been made to use many such techniques,
approving it for publication was Yongjie Li. including identifying deficiencies in the farm produce and

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 10, 2022 41525
 K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

several other critical areas. In [6], color and pattern anal- We report the design of a health monitoring system of
ysis applied to identify multiple deficiencies in paddy leaf the plants which is based on the calculation of the Normal-
images have been reported. There are also attempts to use ized Difference Vegetation Index (NDVI) to distinguish the
internet of things (IoT) platforms and cloud computing as healthy and non-healthy plants using images that are captured
aids to farming. Authors in [7] proposed a framework that by a near infra-red (NIR) camera as part of an IoT set-up and
combines cloud computing and unified IoT for application in integrated with a learning-aided platform. The low-cost pack
the agriculture sector. IoT has also been a preferred option connecting several sensors and the NIR camera forms the IoT
for water flow control in agriculture [8]. It has been com- and revolves around a programmed microcontroller linked
bined with pattern analysis techniques for providing smart to a computer system and subsequently to a cloud server.
agriculture solutions [9], including classification of weeds Subsequently, the entire framework works together with a
[10] and crops disease detection [11], [12]. The applica- learning-based system that identifies diseased and healthy
tion of AI-based methods for predictive farmland optimiza- leaves. The framework is formulated after conducting a series
tion has been reported in [13], a combination of IoT and of experiments involving clustering, SVM, machine, and
machine learning tools for goat monitoring in [14], soil deep learning methods. The work involves experiments with
texture classification using support vector machine (SVM) two clustering-based segmentation methods: K-means clus-
by Barman et al. in [15], chlorophyll detection by a class of tering (KMC) and Fuzzy C-means (FCM) clustering used to
learning techniques explored in [16], etc. The performance of separate healthy and diseased leaf segments. The specific AI
deep learning methods for agriculture has been highlighted tools used are SVM, a Multi-Layer Perceptron (MLP) that is
in [17]–[20]. Application of AI based method in combination a feed-forward ANN, a Time Delay Neural Network (TDNN)
of spatio-temporal methods for real life situations have been that is another feed-forward ANN, an Adaptive Neuro-Fuzzy
reported [21], [22], [26]–[28]. These are some of the recent Inference System (ANFIS) that is a fuzzy-based decision-
works related to the use of AI and IoT tools in agriculture and making method, and a Convolutional Neural Network (CNN)
certain real-life situations. that is a popular deep learning tool. The AI aspect of the work
The discussion above establishes the importance of is deployed in a cloud server which can be accessed using
learning-based tools and IoT in the agriculture sector. an app. After extensive on-field trials, it has been found that
As health monitoring of the plants is one of the foremost the system is useful for a class of agricultural produce com-
tasks in smart agriculture, there are ample scopes to explore monly seen in hot and humid conditions of India. Especially,
innovative ways to make the farmer’s life easier, especially the deep learning-based decision support system formed
in a country like India where agriculture and allied sectors by the VGG16 CNN proves to be effective when used in
employ a huge section of the population. Here we describe the combination with Gray level co-occurrence matrix (GLCM)
design of a proof-of-concept approach to see the effectiveness features and FCM based clustering (which performs seg-
of an arrangement based on multiple sensors and AI based mentation of the region of interest (RoI) of tomato leaves)
decision support system used for process monitoring and con- and adopted as part of the health monitoring system. The
trol for improving yield of agricultural produce. The proposed system is also used to identify effective usable vegetation
system is part of a precision farming approach where the cover of farmland. The novelty of the system is the design
application of IoT devices, image processing techniques and of a deep learning-based decision support system formed by
an AI framework are configured to derive decision states and the VGG16 CNN which proves to be effective when used
execute process control to help the farmer to maximize the in combination of GLCM features and FCM based cluster-
produce. In this work, the decision states obtained from the ing (which performs segmentation of the region of interest
AI system regarding plant heath triggers a pump for water (RoI) of tomato leaves) and adopted as part of the health
sprinkling. The farmer maybe constrained to keep an eye monitoring and process control system which is also used
continuously on a plot of land where certain crops are being to identify effective usable vegetation cover of a farmland.
grown. So, for continuous monitoring and deriving decision The rest of the paper is organized as follows. In Section 2,
states regarding plant health such a set-up is considered to be the proposed work is discussed in detail. The results and
effective. It provides an approach to automate certain repeti- discussion have been covered in Section 3. The outcomes
tive tasks and assists the farmer to execute healthy practices of the work have been summarized in the conclusion
for obtaining better agricultural yield. The IoT system is section.
required to capture the state of the geographical elements and
facilitate continuous monitoring. A drone might be an exten- II. PROPOSED WORK
sion of an IoT system. The Landsat images used currently Here, we discuss the details of the plant leaf health monitoring
to validate the ability of the AI system. The Landsat images system deployed over an IoT, a cloud server and the decision
can be replaced by real-time feed obtained from a camera states generated using a class of clustering, SVM and learning
mounted on a drone. Ground truth is that the life of the farmer aided methods. Before discussing the design and working
is very miserable. So, any arrangement that makes the misery of the complete system, some basic notions related to the
of the farmer less is always a healthy development. It matters different elements of the work are discussed in the sections
a lot for populous countries like India. below.

41526 VOLUME 10, 2022

K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

(LBGLCM), Gray Level Run Length Matrix (GLRLM),

and Segmentation-based Fractal Texture Analysis (SFTA)
[24]. Further, GLCM forms a composite set using features
like skewness, standard deviation, homogeneity, contrast,
smoothness, correlation, kurtosis, energy, entropy, mean,
variance, and root mean square (RMS) to make the extracts
content-rich. The GLCM feature determines the textural rela-
tionship among pixels and is related to the second-order
statistics. The GLCM features determines how gray scale
intensities are distributed vertically and horizontally and also
in different orientations along the diagonals. With the help
of the GLCM features certain statistics like contrast, cor-
FIGURE 1. Generic IoT setup.
relation, energy and homogeneity can be calculated which
are useful in determining the texture of the image. The LBP
features denote statistical and structural model of the tex-
A. THEORETICAL CONSIDERATIONS
tual content of images and demonstrate high invariance to
In this section, we include brief discussion on some of the
light intensity variations. The LBGLCM feature is generated
theoretical considerations of the techniques related to the
by combining LBP and GLCCM algorithms. The GLRLM
work. We specially highlight the key aspects of NDVI, IoT,
feature is obtained from texture representations that con-
clustering methods, ANN techniques like MLP and TDNN,
tain each pixel’s spatial plane features connected to certain
ANFIS, SVM and CNN.
higher-order statistics. The SFTA algorithm carries out a
fractal-based analysis of the image texture. Skewness is asso-
1) NDVI
ciated with a third-order moment and represents a sample’s
NDVI is used to differentiate between bare soil and grass deviation from an ideal probability distribution. Standard
or forest and differentiate between various crop stages etc. deviation represents the deviation of a sample’s statistical
It works using the principle that healthy plants reflect more properties from its mean. Homogeneity signifies the variation
near infra-red (NIR) and green light compared to other a sample may have from its average statistical distribution.
colours. So, more is the chlorophyll (healthy sign), more Contrast denotes the pixel intensity variations with regard to
will be NIR reflection [16]. NDVI is a ratio between the red the neighborhood. Smoothness ensures the absence of sudden
(R) and near infrared (NIR) values and is expressed as variation of pixel intensity.
NIR − R Correlation is linked with the similarity of pixel variations
NDVI = (1)
NIR + R between two different images. It can measure the changes in
the pixel intensity within the image compared to the situation
2) IOT when the variations had not taken place. Kurtosis is a fourth-
IoT is meant as a collection of physical devices, sensors, order statistic. Energy is associated with localized changes
communication systems, user interfaces, etc., in a connected in an image. Entropy denotes the information richness of an
state used to provide analytics, process control and real-time image. RMS outlines the texture variations occurring in an
decision support in a range of situations and applications [23]. im-age and captures the distortions in statistical distributions.
A generic IoT setup is shown in Fig. 1.
5) ANFIS
3) CLUSTERING METHODS
The ANFIS is based on the Sugeno-type fuzzy systems
These are a type of unsupervised grouping method where,
and can be implemented using a multilayered feed forward
depending upon some similarity, objects are placed in seg-
structure [25] as shown in Fig. 2.
ments or classes. Two types of clustering methods are used.
First one is the K-means clustering (KMC) and the second
one is the Fuzzy C-Means clustering (FCMC) [5].

4) FEATURE EXTRACTION METHODS

Features provide a concise and relevant description of an
input image and assist in appropriate decision-making using
classifiers. However, a deep learning network may not require
features as feature learning is part of such networks. But
other learning-based classifiers require certain features. For
agriculture produce and plants, a few popular features are
Gray level co-occurrence matrix (GLCM), Local binary pat-
tern (LBP), Local Binary Gray Level Co-occurrence Matrix FIGURE 2. ANFIS structure.

VOLUME 10, 2022 41527

 K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

In Layer 1 (fuzzification), each node generates the mem-

bership grades of a linguistic label. Here a membership func-
tion of the type Gaussian Membership Function (MF) is
used. The Aj and Bj are the linguistic variable with Gaussian
MF. In Layer 2 (antecedent), using any other fuzzy AND
operation, the nodes calculate the firing strength of each
rule. Next, in Layer 3 (normalization), the nodes find out the
normalized firing strength. It calculates ratios of the rule’s
firing strength to the sum of all the rules firing strength. The
nodes of Layer 4 (consequent) compute consequent parame-
ters which are dependent on Layer 3. It is an adaptive layer.
In the Layer 5 (aggregation), the overall output is calculated
where each node aggregates the final output as the sum of
all incoming signals. The layers are linked by connectionist
links which are adaptive and update during each cycle of
learning. The output is compared with the desired goal and
the updating of the weights which multiply the inputs to each
layer is continued till the objective is attained.
FIGURE 3. Layout of the MLP.

6) MLP
The MLP is a multi-layer feed forward ANN which is made
up of one input and one output layer and one or several
hidden layers with log-sigmoid, tan-sigmoid or purely linear
activation functions. A generic MLP architecture is shown in
Fig. 3 [23]. Let X be an input applied to a MLP. Let W1ij be
the weight matrix between input and hidden layer and W2jk
be the weight matrix required to connect the artificial neurons
between the middle layer and the output layer. Here, i keeps
count of number of input samples, j is related to the indexing
of the number of hidden layer neurons and k is linked to the
output layer size. The output of the MLP is expressed as
XK XJ XI
Yo = fk { (X i W1ij + bi )W2jk (2)
k=1 j=1 i=1
Let YT be the target output. The mean square error (MSE) is FIGURE 4. Layout of the TDNN.
expressed as:
1 XP p 2
method, the TDNN is able to track variations in data due
E= (Yop − YT ) (3) to time. Like MLP, it also has multiple layers with input,
2 p=1
output and hidden types and is trained with back propagation
During training, weight update happens following a gradient
algorithm. In our case with have used 1 to 2 numbers of
descent principle. The weight update expression is given as
positive delay to develop time tracking ability of the ANN.
1E Fig. 4 shows a layout of the TDNN [23]. For the TDNN, the
W [n + 1]j = W [n]j + µ (4)
1W [n] output and other expressions are same as shown in (2)-(4)
where µ is the learning rate of the network usually taken to except that the input shall be a combination of X (n)+X (n+T )
be a fraction between 0 and 1. The MLP is trained with back where T denotes the delay used in the input feed layer.
propagation algorithm till it learns the patterns completely.
The choice of hidden layers’ number and the activation 8) SUPPORT VECTOR MACHINE (SVM)
functions depends upon the requirements. In our case we The SVM is a supervised learning-based approach used for
have used a one hidden layer MLP with one and half times classification and regression. The working of the SVM is
more numbers of artificial neurons of the input layer. These explained using Fig. 5. The discrimination boundary between
activation functions are of log-sigmoid type and are used to the classes is laid by
learn the leaf classification patterns.
H0 : w.x i + b = 0 (5)
7) TIME DELAY NEURAL NETWORK (TDNN) where input x i is scaled by weight w and aided by the
The TDNN is another multi layered feedforward ANN like bias b. It classifies the patterns according to the values asso-
MLP but has delayed feed in the input. With this feeding ciated with support vectors across the two sides of the

41528 VOLUME 10, 2022

K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

FIGURE 5. Working of the SVM.

FIGURE 7. Block diagram of the proposed design.

Its hardware elements include a Raspberry Pi 3 microcon-

troller, temperature and humidity sensor (DHT11), soil mois-
ture sensor, motor pump with a relay, and a NIR camera.
FIGURE 6. Basic layout of CNN.
Further, an MQ-135 gas sensor pack is also connected to
detect spurious gases and initiate certain process control. The
sensors and the camera are connected to the microcontroller
hyperplane [5]. Here, each sample value is considered as a to form an IoT pack connected to a host computer and
point in n-dimensional space with the numerical count of each subsequently to the internet so that seamless data transfer,
pattern representing a particular coordinate. The closeness of web access, and related processing can occur. There are two
the coordinate to the class-region derived through the use of aspects to the design. First, an on-field feed mechanism as
support vectors determines the classification of the samples. part of which data regarding the leaf of a plant using a NIR
In this work, the Gaussian kernel is used for passing the input camera, moisture content, and temperature of the surround-
x i and generating multi-class states which are subsequently ings are connected using the IoT pack. Next, a user can use a
scaled by the weight w for performing the classification. The smartphone application (designed for the purpose) to feed the
SVM is used as a benchmark technique in this work. related data online to the system. Leaf health of certain com-
mon agricultural produce like cabbage, chilly, cauliflower,
9) CONVOLUTIONAL NEURAL NETWORK (CNN) carrot, coriander, cucumber, pumpkin, ginger, potato, tomato,
The CNN is a deep learning network used for pattern classi- ladyfinger, and gourd are taken into consideration.
fication and is constituted by an input, output and a number Initially, through certain on-field visits, training samples
of middle layers. These include multiple repetitions of max- covering healthy, diseased, and dead leaves numbering at
pooling layer, sub-sampling layer, normalization layer, fully least 1000 of each of the twelve vegetable types are cap-
connected layer etc. before connecting to a classification tured using the NIR camera. Data bootstrapping techniques
layer which is normally the output layer (see Fig. 6). The (Fig. 8) are used to increase each type’s training and testing
CNN doesn’t require much of preprocessed and labelled data set volumes when required. Simultaneously the reading of the
while feature learning takes place layer by layer without any temperature, moisture, and the surrounding gaseous contents
human intervention. The frontend has a number of convolu- are recorded both using the IoT pack and manually. The
tional masks of varying sizes which take part in data capture NDVI value for each type of leaf sample is calculated and the
with minute details which greatly enhance its discrimina- health state labeled. Accordingly, the associated temperature,
tion and internal hierarchical feature extraction contributing moisture, and gaseous surrounding attributes are embedded
towards its superior classification capabilities [5]. into a file used with the cloud-based learning system. These
physical parameters are required for process control like
B. SYSTEM MODEL AND WORKING running a water pump etc., as per the vegetation require-
The system model of the proposed design is shown in Fig. 7. ments. But the main focus is on designing a learning-based
The system has both hardware and software components. system for monitoring plant health. Here we have explored

VOLUME 10, 2022 41529

 K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

FIGURE 8. Bootstrapping method to increase data samples for training.

FIGURE 9. Process logic of plant leaf classification.

the effectiveness of SVM, MLP, TDNN, ANFIS, and CNN
in carrying out the training and subsequently providing clas-
TABLE 1. Experimental parameters.
sification for different leaf states for the twelve identified
types of vegetables. The benchmark experiments are carried
out with the tomato plant. The NDVI values are calculated
using certain steps. First, using the NoIR camera, a visible
leaf of a plant is captured. Next, the NoIR image is split
into R, G, B, and NIR planes from which NDVI values
of individual planes are calculated. An average of all these
values is taken from this representation, which becomes the
required NDVI value. For a healthy leaf, the NDVI value is
0.96, the diseased has a value of 0.42, and a dead leaf has
a value of −0.02. These are taken for apriori classification
and labeling of the leaf samples. Now taking the values of the
sensor, separate training cycles are carried out for each of the
selected learning-based systems.
The updated states of the learning along with the classifi-
cation decision details are held in a database. An application
for the user has been designed using the Blync app to hold
the data coming from the field sensors and interfaces in a
cloud server. The images of the plant leaves are taken, prepro-
cessed (for noise removal, enhancement, etc.), and subjected
to segmentation. Clustering techniques like KMC and FCM
are used for this purpose. Next, features using GLCM, LBP,
LBGLCM, GLRLM, and SFTA are extracted and applied
to SVM, MLP, TDNN, ANFIS, and CNN for carrying out
the classification. Further, features like skewness, standard
deviation, homogeneity, contrast, smoothness, correlation, III. EXPERIMENTAL DETAILS AND RESULTS
kurtosis, energy, entropy, mean, variance, and root mean Extensive experiments have been carried out to check the per-
square (RMS) are considered a composite set derived from formance of the proposed design. The relevant experimental
the GLCM and used extensively to make the process robust. specifications are summarized in Table 1.
The processing logic is summarized in Fig. 9. The experiments are carried out taking into account these
Further, the learning-based systems use the sensor feeds experimental specifications covering pre-processing method,
to automate the process of watering the plants keeping an segmentation approach, features considered, classifiers used,
eye on threshold set. The sensors continuously provide the parameters necessary and details of data applied to train the
readings regarding the temperature and the humidity sensor of learning aided decision support system. A GUI has been
the surroundings and the soil moisture state. The combination developed for applying the samples into the system. The
of these three readings is used to decide upon the triggering outputs of each stage can be obtained using the GUI while
of the water pump and the duration. All probable conditions the system processes and generates responses. After carrying
involving the temperature, humidity and soil moisture and out the pre-processing operations like noise removal and
association with the watering requirements are ascertained enhancement (Fig. 10), and re-sizing in some cases, image
beforehand for a particular crop type and applied to the segmentation is carried out using two clustering techniques
learning system. These time dependent sensor feeds are learnt which have already been mentioned above.
and the mapping with the watering states are established by Weighted moving average filtering provides 8 to 12%
the learning systems. improvement in terms of peak signal to noise ratio (PSNR)

41530 VOLUME 10, 2022

K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

TABLE 2. Accuracy (Acc), IOU and PA performance of KCM and FCM clustering approaches used for segmentation.

values compared to spatial filtering during noise removal.

Similarly, histogram processing consistently yields 9% better
results in terms of PSNR compared to enhancement oper-
ations using filter masks. The fuzzy based segmentation
approach related to FCM provides better segmentation per-
formance (segmented portion + GLCM feature + classifier)
as observed from the experimental results (Table 2).
This is because of the fact that the randomness and minor
variations in the pixels of the leaf samples are handled well
by the fuzzy attributes in the approach. The classifiers pro-
vide outputs in terms of true positive (TP), true negative
(TN), false positive (FP) and false negative (FN). The ratio FIGURE 10. Image processing operations over a diseased leaf.
between the sums of TP and TN (TP+TN) and TP, TN,
FP and FN (TP+TN+FP+FN) in percentage gives accuracy.
Further, as the segmentation block plays a very significant consistent accuracy and demonstrates the best ability to han-
role in determining the reliability of the proposed system, dle variations in the input. The SVM takes least amount of
its performance is ascertained in terms of Intersection over time to train but reliability performance with samples not
union (IOU) and Pixel Accuracy (PA). The IOU and PA within the extended data set is the worst amongst the methods
values are obtained using considered for the work. The multi layered feature transform
taking place in case of the CNN based VGG16 model enables
TP
IOU = (6) detailed capture of the relevant content and hence provides
TP + FP + FN the best results despite a higher requirement of training time.
TP
PA = . (7) However, the testing time for all the methods are nearly equal.
TP + FN The model has a classifier layer at the end which is pre-
The average values of IOU and PA obtained from both ceded by a block formed by three fully connected layers two
the segmentation methods used with the classifiers are shown of which have a length of 4096 and the last layer has a length
in Table 2. Moreover, the clustering-based approach is used of 1000. The classification layer is formed by a soft-max layer
for selecting the region of interest (RoI) which is reinforced with log-sigmoid activation function. The main body of the
using manual labelling. The clustering based RoI selection VGG16 CNN is formed by five repetitions of two blocks of
can be used to automate the class labelling process. This is 3 × 3 convolution mask and a layer of 64 rectified linear
shown in Fig. 11. The GLCM algorithm enables calculation unit (ReLU) activations followed by a 2 × 2 max-pooling
of contrast, correlation, energy, homogeneity, mean, standard layers. This thick block carries out the feature learning from
deviation, entropy, RMS, variance, smoothness, kurtosis and the input samples. Experiments are carried out using the fea-
skewness values (Fig. 12) which are applied to a benchmark ture set and also by feeding the images directly. The soft-max
SVM classifier. The classifier not only classifies in terms of layer is initially provided with labelled class data for carrying
healthy and unhealthy leaves but also detects disease attacks out the learning. In another set of experiments the approach
like bacterial spot, blight, septoria, spider mites and yellow is made automated by using the FCM to generate the RoI
leaf curl. These are common disease associated with tomato from the leaves which are used as targets for classification.
plants. Fig. 14 shows a training time accuracy performance generated
The classification has also been carried out with MLP, using the feature set.
TDNN, ANFIS and CNN. In case of the deep learning-based This accuracy (number of correct pixel classification com-
recognition, the CNN based VGG16 model is taken for the pared to the total number of pixels of an image sample
purpose of classifying the leaf disease. The configuration of expressed in percentage) of around 95% (Fig. 14) in the case
the VGG16 is shown in Fig. 13. of tomato leaf samples for the five different disease types
The maximum time consumed by each of the learn- is generated after seven epochs of training with a batch size
ing based methods while training with the full comple- of 50 which is repeated for about 500 training samples. The
ment of training data is shown in Table 1. While the average performance with all the identified plant types and
CNN based VGG16 model takes the maximum time, it is class groupings suffers a bit but is over 94%. When the images
robust, doesn’t require additional feature learning, provides are used as input, the training takes over ten times more

VOLUME 10, 2022 41531

 K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

FIGURE 11. GUI showing pre and post pre-processing, segmentation and
selection of region of interest operations over a diseased leaf.

FIGURE 13. VGG16 CNN used for plant leave disease.

FIGURE 12. Feature values of disease affected region from GLCM

algorithm.

cycles, but accuracy performance is improved to 96% with

no dedicated feature extraction block requirements. The per-
formance obtained with the validation set is better compared
to the training set. This is because the validation data set
has better diversity than the training dataset. During training,
the AI system completely becomes familiar with the training
data set. Hence, the training dataset loses diversity, but the
validation dataset is formed using bootstrapping technique
which has greater diversity. Learning-based systems work
well with datasets where there is diversity. The experimental FIGURE 14. Training time performance of the VGG16 CNN.

results of the various experiments are explained below.

The accuracy performance of the segmentation block is fuzzy-based approach of capturing the segment details helps
shown in Table 2. The KCM and FCM approaches are to extract the segments better. The use of the fuzzy-based
used to extract out the leaf segments of the tomato leaf image segmentation also helps in finer discrimination even if
samples. For these segments, features are extracted using there might be a very small amount of variation for extraction
the GLCM approach and classification using SVM, MLP, out the leaf’s healthy and diseased sections, which adds to the
TDNN, ANFIS, and CNN classifiers. Except for ANFIS in improvement in the performance of the classifiers.
all the classifiers, the FCM based segmentation contributes Another set of results are generated to show the effective-
towards improved performance. In the case of CNN, there ness of the features. We have used GLCM, LBP, LBGLCM,
is a 2% improvement while in cases of SVM, MLP and the GLRLM, and SFTA features to derive decisions regarding
TDNN there is 1% better result due to the use of the FCM. The the health state of the plants. For about 500 tomato leaf

41532 VOLUME 10, 2022

K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

TABLE 3. Accuracy performance of feature sets with a few identified

classifiers as part of health monitoring of tomato leaf samples.

FIGURE 15. Red band and NIR band images obtained as part of
vegetation estimation.

TABLE 4. PDR with varying pack sizes of sensors with Bluetooth, physical
and Wi-Fi connectivity.

samples sets of five different features, their combinations are

extracted and applied to SVM, MLP, TDNN, ANFIS and
CNN classifiers. This is calculated for a number of trials for
the samples used during training and testing. The average
accuracy performances are recorded and are shown in Table 3.
The healthy and diseased extracts segmentation is done
using the FCM approach, showing improved performance,
as noted in Table 2. It is seen that the best performance
with GLCM is obtained from CNN, giving 94% accuracy,
while with SVM, MLP, TDNN, and ANFIS, the variations
are between 84 to 87%. With LBP features, there is minor Landsat images can be replaced by real-time feed obtained
performance improvement with SVM, MLP, TDNN, and from a camera mounted on a drone. The VGG16 network is
ANFIS but no change with the CNN. There is a minor trained with Landsat satellite data to classify thick vegetation,
performance fall with LBGLCM and GLRLM features with normal vegetation, and vegetation cover less soil, water, and
SVM and MLP classifiers compared to LBP, while with others (not falling into these four). NVDI threshold values
TDNN, ANFIS and CNN, there are no changes in accuracy. are calculated for use with the system to derive the class
The SFTA features show minor improvement in accuracy decisions for these classes. From the satellite images band4
with SVM and MLP classifiers, but the performance is con- (Red channel) and band5 (Near Infrared band) (Fig. 15),
stant with TDNN, ANFIS, and CNN. A few combinations contents of a particular area are generated, which are next
of features are explored, out of which GLCM + GLRLM, used to calculate the NDVI values.
GLCM+SFTA, and GLCM+GLRLM+SFTA show varia- The NDVI image of an area is obtained, which forms the
tions in performances, as shown in Table 3. The CNN and basis of estimating the total vegetation cover of the area. The
in many cases the ANFIS show very little variation in perfor- image dataset consists of 9627 samples separated into a 60:40
mance despite changes in the features and the use of combina- ratio where 60% of them have been used for training, and
tions. Since the GLCM feature with FCM based segmentation 40% applied for model evaluation. With a batch size of 50,
and classification using CNN gives the best performance, this the model took seven epochs completed over 35 seconds on a
combination (FCM+GLCM+CNN) is used as the standard cloud-based GPU. In this effort, the weights and the features
combination for the subsequent work. of the datasets are obtained through different processes which
As the proposed system is intended to help the farmers, amounted to over two hours of computation. A separate val-
an additional characteristic incorporated into it is its ability to idation set generated using bootstrapping method is applied
identify farm vegetation and the geographical distribution of for validating the VGG16 network after training. The output
the surroundings. Most agricultural lands are surrounded by a of the FCM+NVDI+CNN combination for a set of experi-
range of vegetation and other geographical elements. During ments related to vegetation estimation is shown in Fig. 16.
rainy seasons, the distribution of thick vegetation, normal Certain experiments have been carried out to see the effec-
vegetation, vegetation cover less soil, water, and others (not tiveness of the IoT arrangement. Experiments involving the
falling into these four) vary. Certain weed types grow very packet delivery ratio (PDR) with sensor pack sizes varying
fast and harm agricultural produce. With it comes insects and between 4 and 16 and connected via Bluetooth, Wi-fi, and
other life forms. So, the distribution of geographical elements physical connection have been carried out. There are one
around farmland and its monitoring using an IoT system is temperature and one moisture sensor in each sensor pack and
essential for the effective management of farmland. The IoT a NoIR camera connected to a Raspberry Pi microcontroller.
system is required to capture the state of the geographical This arrangement is for one plant. The system’s effective-
elements and facilitate continuous monitoring. A drone might ness has been tested by connecting the sensor packs to the
be an extension of an IoT system. The Landsat images are controller computer in numbers between 4 and 16 to form a
currently used to validate the ability of the AI system. The robust IoT arrangement. The effectiveness of the arrangement

VOLUME 10, 2022 41533

 K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

TABLE 5. Outcomes of statistical trials involving recall, specificity, type I and type II errors, F1 score, micro F1 and macro F1 values involving leave disease
recognition (C1) with GLCM features and FCM segmentation and geographic distribution identification (C2) using SVM, MLP, TDNN, ANFIS and CNN
classifiers.

TABLE 6. Summary of results and comparison.

FIGURE 16. Percentage distribution of different classes of geographic

content extracted as part of vegetation estimation.
reported. The samples are divided into a 60:40 ratio, with 60%
used for training and 40% used for testing. Samples have light
intensity variations, and noise-related distortions assumed to
is checked in terms of PDR, and the results are summarized be identical to those seen due to faulty sensor pickup. The
in Table 4. The physical connectivity has the best PDR, while training time associated with the VGG16 CNN while feeding
Bluetooth links prove to be less reliable. the complete image and with no feature is considerably higher
We have performed certain statistical experiments to ver- compared to the case when features are used. The difference
ify the experimental reliability and real-time robustness in is at least two times less in the case when features are used.
on field conditions. The results are summarized in Table 5. But when no features are used, the associated computation
It includes the outcomes of statistical trials involving recall, cycles are saved, and the design complexity is lowered. How-
specificity, type I and type II errors, F1 score, micro F1 and ever, it increases the computational workload of the CNN,
macro F1 values involving leave disease recognition (C1) but if the weights are trained separately, and one cycle is
with GLCM features and FCM segmentation and geographic completed successfully, subsequent iterations become less
distribution identification (C2) using SVM, MLP, TDNN, intensive. The average results clearly indicate the advantage
ANFIS and CNN classifiers. of the proposed approach.
A summary of the performance of the proposed approach
and its comparison with certain previous research is included IV. CONCLUSION
in Table 6. As discussed above, the techniques reported Here, we have discussed the design of a smart agriculture
in [22] and [15] have been reproduced and applied to per- assistance system that is based on IoT and learning-based
form plant health monitoring. These are compared with decision making. The proposed system has a sensor pack
those obtained with pre-trained VGG16 CNN + GLCM and that obtains a readout of temperature, moisture of the air,
VGG 16 CNN taking the complete leaf sample as input (with- and the soil and a camera that captures NIR images of
out features) as discussed above. With around 9000 samples, the plant leaves. The sensor readings are used to obtain
the experiments are carried out, and the average results are continuous monitoring and to regulate a small-scale water

41534 VOLUME 10, 2022

K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

sprinkling mechanism. Further, the images obtained from the [12] A. Devaraj, K. Rathan, S. Jaahnavi, and K. Indira, ‘‘Identification of plant
NIR camera are used to train a set of learning-based tools disease using image processing technique,’’ in Proc. Int. Conf. Commun.
Signal Process. (ICCSP), Apr. 2019, pp. 749–753.
that help monitor the health of the plant leaves. We identify [13] M. O. Adebiyi, R. O. Ogundokun, and A. A. Abokhai, ‘‘Machine learning–
twelve types of vegetation for the study, out of which five based predictive farmland optimization and crop monitoring system,’’
disease types of the tomato leaves are categorized into five Scientifica, vol. 2020, pp. 1–12, May 2020.
[14] Y. Rao, M. Jiang, W. Wang, W. Zhang, and R. Wang, ‘‘On-farm welfare
different types using a pre-trained VGG16 CNN. The work monitoring system for goats based on Internet of Things and machine
includes experiments conducted with two clustering-based learning,’’ Int. J. Distrib. Sensor Netw., vol. 16, no. 7, pp. 1–17, 2020.
segmentation methods, namely KMC and FCM, which sep- [15] U. Barman and R. D. Choudhury, ‘‘Soil texture classification using multi
class support vector machine,’’ Inf. Process. Agricult., vol. 7, no. 2,
arates the healthy and diseased leaf segments and features, pp. 318–332, Jun. 2020.
including GLCM, LBP, LBGLCM, and GLRLM SFTA. Spe- [16] U. Barman and R. D. Choudhury, ‘‘Smartphone image based digital chloro-
cific AI tools like SVM, MLP, TDNN, ANFIS, and the phyll meter to estimate the value of citrus leaves chlorophyll using linear
regression, LMBP-ANN and SCGBP-ANN,’’ J. King Saud Univ. Comput.
VGG16 CNN are trained to provide automated decision Inf. Sci., doi: 10.1016/j.jksuci.2020.01.005.
support for the classification of the plant leaf disease types [17] P. Sharma, Y. P. S. Berwal, and W. Ghai, ‘‘Performance analysis of deep
through a cloud server that can be accessed using an app. learning CNN models for disease detection in plants using image segmen-
tation,’’ Inf. Process. Agricult., vol. 7, no. 4, pp. 566–574, Dec. 2020.
Extensive on-field trials show that the system is effective [18] S. H. Lee, H. Goëaua, P. Bonneta, and J. Alexis, ‘‘New perspectives on
in hot and humid conditions, especially the FCM, GLCM plant disease characterization based on deep learning,’’ Comput. Electron.
feature, and the VGG16 CNN proves to be reliable. Agricult., vol. 170, Mar. 2020, Art. no. 105220.
[19] Z. Zhai, J. F. Martínez, V. Beltran, and N. L. Martínez, ‘‘Decision support
Further, the system has been found to be effective in systems for agriculture 4.0: Survey and challenges,’’ Comput. Electron.
identifying certain usable vegetation cover of farmland and Agricult., vol. 170, Mar. 2020, Art. no. 105256.
regulation the watering mechanism of crops. The system [20] T. V. Klompenburga, A. Kassahuna, and C. Catalb, ‘‘Crop yield prediction
using machine learning: A systematic literature review,’’ Comput. Electron.
proves to be effective in identifying the disease type of cer- Agricult., vol. 170, Oct. 2021, Art. no. 105709.
tain agricultural products. It proves reliable in discriminating [21] A. Ali, Y. Zhu, and M. Zakarya, ‘‘A data aggregation based approach to
thick vegetation, normal vegetation, vegetation cover less exploit dynamic spatio-temporal correlations for citywide crowd flows
prediction in fog computing,’’ Multimedia Tools Appl., vol. 80, no. 20,
soil, water, and others (not falling into these four) controlling pp. 31401–31433, Aug. 2021, doi: 10.1007/s11042-020-10486-4.
the water spaying mechanism associated with the complete [22] N. Awan, A. Ali, F. Khan, M. Zakarya, R. Alturki, M. Kundi,
framework. An expanded version of the work with certain M. D. Alshehri, and M. Haleem, ‘‘Modeling dynamic spatio-temporal
correlations for urban traffic flows prediction,’’ IEEE Access, vol. 9,
added features shall be an effective aid of the farmers. pp. 26502–26511, 2021, doi: 10.1109/ACCESS.2021.3056926.
[23] S. Greengard, The Internet of Things. Cambridge, MA, USA: MIT Press,
2015.
REFERENCES
[24] Ş. Öztürka and B. Akdemirb, ‘‘Application of feature extraction and
[1] A. D. Foster and M. R. Rosenzweig, ‘‘Microeconomics of tech- classification methods for histopathological image using GLCM, LBP,
nology adoption,’’ in Economic Growth Centre Discussion, vol. 2. LBGLCM, GLRLM and SFTA,’’ in Proc. Int. Conf. Comput. Intell. Data
Annu. Rev. Econ., 2010, pp. 395–424. [Online]. Available: https://www. Sci. (ICCIDS), 2018, pp. 40–46.
annualreviews.org/doi/abs/10.1146/annurev.economics.102308.124433 [25] S. Kaushik and S. Tiwari, Soft-Computing, 1st ed. New Delhi, India:
[2] M. A. Shahin and S. J. Symons, ‘‘A machine vision system for grading MacGraw-Hill, 2018.
lentils,’’ Can. Biosyst. Eng., vol. 43, pp. 7–14, Jun. 2001. [26] A. Ali, Y. Zhu, Q. Chen, J. Yu, and H. Cai, ‘‘Leveraging spatio-temporal
[3] R. Shahzadi, J. Ferzund, M. Tausif, and M. Asif, ‘‘Internet of Things based patterns for predicting citywide traffic crowd flows using deep hybrid
expert system for smart agriculture,’’ Int. J. Adv. Comput. Sci. Appl., vol. 7, neural networks,’’ in Proc. IEEE 25th Int. Conf. Parallel Distrib. Syst.
no. 9, pp. 341–350, 2016. (ICPADS), Dec. 2019, pp. 125–132.
[4] K. Jha, A. Doshi, P. Patel, and M. Shah, ‘‘A comprehensive review [27] A. Ali, Y. Zhu, and M. Zakarya, ‘‘Exploiting dynamic spatio-temporal
on automation in agriculture using artificial intelligence,’’ Artif. Intell. graph convolutional neural networks for citywide traffic flows prediction,’’
Agricult., vol. 2, pp. 1–12, Jun. 2019. Neural Netw., vol. 145, pp. 233–247, Jan. 2022.
[5] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, [28] A. Ali, Y. Zhu, and M. Zakarya, ‘‘Exploiting dynamic spatio-temporal cor-
MA, USA: MIT Press, 2016. relations for citywide traffic flow prediction using attention based neural
[6] M. V. Latte and S. Shidnal, ‘‘Multiple nutrient deficiency detection in networks,’’ Inf. Sci., vol. 577, pp. 852–870, Oct. 2021.
paddy leaf images using color and pattern analysis,’’ in Proc. Int. Conf.
Commun. Signal Process. (ICCSP), Apr. 2016, pp. 1247–1250.
[7] J. Rui and S. Danpeng, ‘‘Architecture design of the Internet of Things
based on cloud computing,’’ in Proc. 7th Int. Conf. Measuring Technol.
Mechatronics Autom., Jun. 2015, pp. 206–209.
[8] V. V. H. Ram, H. Vishal, S. Dhanalakshmi, and P. M. Vidya, ‘‘Regulation
of water in agriculture field using Internet of Things,’’ in Proc. IEEE
Technological Innov. ICT Agricult. Rural Develop. (TIAR), Jul. 2015,
pp. 112–115.
KANDARPA KUMAR SARMA (Senior Member,
[9] A. Kapoor, S. I. Bhat, S. Shidnal, and A. Mehra, ‘‘Implementation of
IEEE) received the M.Tech. degree in signal pro-
IoT (Internet of Things) and image processing in smart agriculture,’’ in
Proc. Int. Conf. Comput. Syst. Inf. Technol. Sustain. Solutions (CSITSS),
cessing from the Indian Institute of Technol-
Oct. 2016, pp. 21–26. ogy Guwahati, Assam, India, in 2005, and the
[10] M. Dyrmann, A. K. Mortensen, H. S. Midtiby, and R. N. Jørgensen, Ph.D. degree in the area of mobile communication
‘‘Pixel-wise classification of weeds and crops in images by using a fully from the Indian Institute of Technology Guwahati.
convolutional neural network,’’ in Proc. Int. Conf. Agricult. Eng., Aarhus, He currently works as a Professor and the Head
Denmark, Jun. 2016, pp. 26–29. of the Department of Electronics and Communica-
[11] J. Amara, B. Bouaziz, and A. Algergawy, ‘‘A deep learning-based approach tion Engineering, Gauhati University, Assam. His
for banana leaf diseases classification,’’ in Proc. Stuttgart, BTW Workshop, research interests include deep learning, mobile
2017, pp. 79–88. communications, speech processing, antenna design, and electronic warfare.

VOLUME 10, 2022 41535

 K. K. Sarma et al.: Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN

KUNAL KINGKAR DAS received the B.Tech. SAMADRITA BHUIYA received the B.Tech.
degree in electronics and communication engi- degree in electronics and communication engi-
neering from Gauhati University, in 2020. He spe- neering from Gauhati University, in 2020, where
cializes in intelligent system design. she is currently pursuing the master’s degree in
microelectronics and VLSI design. She special-
izes in intelligent system design and intelligent
farming.

DMITRII KAPLUN (Member, IEEE) received the

Ph.D. degree, in 2009. In 2009, he defended his
Ph.D. thesis in digital signal processing at Saint
Petersburg Electrotechnical University ‘‘LETI,’’
Saint Petersburg, Russia. He was an Associate Pro-
fessor, in 2015. He is currently a Senior Researcher
at Saint Petersburg Electrotechnical University
‘‘LETI.’’ His current research and academic work
VIKASH MISHRA received the B.Tech. degree in are concerned with digital signal and image pro-
electronics and communication engineering from cessing, radio monitoring and hydroacoustic mon-
Gauhati University, in 2020. He specializes in itoring applications, embedded and reconfigurable systems, computer vision,
intelligent system design and artificial intelli- and machine learning. He regularly takes part in different collaborative
gence. He is currently working as an Assistant projects related to signal processing, neural networks, and software-hardware
System Engineer with Tata Consultancy Service, implementation of digital signal processing algorithms. The most substantial
India. results are in the fields of digital signal and image processing, embedded
systems, neural networks, and machine learning. He is the author of more
than 50 papers in journals and conference proceedings.

41536 VOLUME 10, 2022