applied

sciences
Article
Using Machine Learning to Grade the Mango’s
Quality Based on External Features Captured by
Vision System
Nguyen Truong Minh Long and Nguyen Truong Thinh *
Department of Mechatronics, HCMC University of Technology and Education,
Ho Chi Minh City 700000, Vietnam; [email protected]
* Correspondence: [email protected]; Tel.: +84-9036-675-673




Received: 27 July 2020; Accepted: 17 August 2020; Published: 20 August 2020


Abstract: Nowadays, mangoes and other fruits are classified according to human perception of low
productivity, which is a poor quality of classification. Therefore, in this study, we suggest a novel
evaluation of internal quality focused on external features of mango as well as its weight. The results
show that evaluation is more effective than using only one of the external features or weight combining
an expensive nondestructive (NDT) measurement. Grading of fruits is implemented by four models
of machine learning as Random Forest (RF), Linear Discriminant Analysis (LDA), Support Vector
Machine (SVM), and K-Nearest Neighbors (KNN). Models have inputs such as length, width, defect,
weight, and outputs being mango classifications such as grade G1 , G2 , and G3 . The unstructured data
of 4983 of captured images combining with load-cell signals are transferred to structured data to
generate a completed dataset including density. The data normalization and elimination of outliers
(DNEO) are used to create a better dataset which prepared for machine learning algorithms. Moreover,
an unbiased performance estimate for the training process carried out by the nested cross-validation
(NCV) method. In the experiment, the methods of machine learning have high accurate over 87.9%,
especially the model of RF gets 98.1% accuracy.

Keywords: fruit grade; image processing; captured images; mango classification; machine learning;
sorting algorithm; KNN; SVM; RF; LDA

1. Introduction
Mango is a popular fruit of Asia. There has been a growing demand for high-quality mangoes in
markets. Therefore, grading the quality of fruit has become vitally important for farmers. Besides,
the quality of mangoes is not only affected by the growth and maturation before harvesting but also
depends on the post-harvest. Manual grading and classifying of mangoes is laborious, resulting in
the cost of fruits increasing and quality being uneven. Increasingly, vendors are placing requirements
not only on external factors like size, color, and firmness but on internal quality factors such as sugar
content and acidity. Although consumers buy fruit based on their external appearances, the taste of the
fruit is used to determine whether the consumer buys again. Therefore, in this study, we use Machine
Learning (ML) to grade the mango’s quality based on external features taken from the computer vision
system combined with weight. The internal quality of the fruit is determined by various measurement
techniques based on non-destructive testing (NDT) [1]. This technique has many advantages but it still
has certain limitations such as poor reliability and the need for constant recalibration. The problem of
mango evaluation has been of concern in many studies. Most of them used both numerical approach
and analytical approach based on external features of mangoes such as color, dimensions, shape,
weight, or defects that extracted from image data. Many authors used different algorithms of ML

Appl. Sci. 2020, 10, 5775; doi:10.3390/app10175775 www.mdpi.com/journal/applsci

Appl. Sci. 2020, 10, 5775 2 of 25

for evaluating the quality of mango like as a new trend in agriculture. Nandi et al. [2] proposed
an ML technique for sorting mangoes in terms of maturity. The captured images from the camera
are converted to the binary images and then the sizes of mangoes is estimated. These features were
used to grade mango into four groups based on the Support Vector Machine (SVM) method. Pise,
Dnyaneshwari, and G.D. Upadhye [3] graded mango by determining the mango’s maturity as well as
quality in the form of size, shape, and surface defects. Experiments of Pandey et al. [4] considered
600 mango samples from several orchards. The quality of mango was decided by its grading standard.
The classification related to distinguishing healthy or diseased mangoes, ripe or mature. Therefore,
the color was used to classify fruits into different categories. Several other studies also mentioned the
estimated volume of mango based on the cylinder approximation analysis method [5]. There were also
some researches using both SVM and Discriminant Analysis (DA) to analyze and estimate the shape
of mango. Thereby, weights of mango were determined in order to classify them [6]. The volume
data were analyzed through a variety of methods. The mangoes were also classified based on their
sweetness [7] which was measured with Near-Infrared Diffuse Reflectance. The quality of mangoes
was also graded by maturity. Nandi et al. [8] graded the quality of mangoes based on RFE-SVM and the
Multi-Attribute Decision Making (MADM) approaches. Besides, image processing was used to extract
the color, size, sphericity, and weight, which was possible to know the maturity time of the mangoes
in each stage as well as grading mango [9]. Procedures [2–6] performed well in mango classification
based on weights and sizes of them. Color and defect were concerned [8,9].
In general, the combination of all the features to classify mangoes is necessary. This brings more
accuracy for evaluation of mango quality. This paper proposes a novel mango grading system based
on external features taken from the computer vision system combined with weight. In computer
vision, a camera is used to capture the external features of mango and the weight is taken from the
load-cell sensor. Furthermore, the algorithm can be applied for the other fruits. The main idea of the
proposed method is to use supervised ML to grade the mango based on the combination of appearance
features and density. The efficiency of the proposed method is evaluated through a small labeled
dataset acquired randomly from several orchards. Supervised ML is the process based on previously
labeled data to give predictions to unlabeled objects. The proposed algorithm is relevant to Multiclass
Classification (MC). Many previous studies presented some effective models in fruit categorization
such as Support Vector Machine (SVM) [3,5,8], Linear Discriminant Analysis (LDA) [5,6,9], Random
Forest (RF) [10], and K-Nearest Neighbors (KNN) [11]. This study, by including four ML models of
SVM, DA, KNN, and RF, is different from previous ones. In reality, density is directly considered as the
factor of quality for a variety of fruits. The contribution of this paper is a mango classification method
which uses fruit density as one of the main factors to classify quality with size and defect. Besides,
density-based methods with external features can be applied to other fruits to evaluate the quality
of the internal quality without complex NDT equipment. The main method based on RF algorithm
is a calculation procedure using the weight and external features extracted from the vision machine.
In addition, three ML models like KNN, LDA, and SVM are applied to grade and classify. On the
other hand, a NCV method is employed to confirm the accuracy of mango classification. Experimental
results show that the system has a high accuracy of 98.1%.

2. Structure of Quality Grading System

This study has been applied and put into practice to experiment in many orchards in Vietnam.
The hardware of the grading system is shown in Figure 1. To acquire the data from captured images
and evaluate geometric parameters of mangoes, several models are employed. The signal of weight is
taken from the load-cell and processed to give weight shown in Figure 2. The signal from the load-cell
is eliminated noise by Kalman filter.
extracted by various methods such as image processing, ML, and deep learning. In this study, the
image processing is more efficient, because of its fast processing time without a large training dataset.
This method changes processing parameters, which are determined more difficult in both ML and
deep learning. The extracted features of the captured image are combined with weight to generate a
full Sci.
Appl. data set10,in5775
2020, order to leverage machine learning models. The second part is the weighting3 ofand
25
grading system to classify mango into G1, G2, and G3.

Figure 1. Mango grading and sorting system used in the study.

In the sorting system, the mangoes enter the vision chamber by a roller conveyor system. During
movement, the mango is rotated around the roller axis and captured at different positions by a
camera to accurately predict its external features. The mangoes then move into the tray conveyor to
weigh and sort them based on the central processing signal. After that, in the central processing unit,
the mangoes’ weight and external features are combined to generate a data set which prepared for
the training in ML models. The unstructured data is converted to structured data, then it is put into
models of ML to predict the grades. Last but not least, the data transmission is carried out on the
server to ensure all processed data in the best.
Figure 1. Mango grading and sorting system used in the study.
Figure 1. Mango grading and sorting system used in the study.

In the sorting system, the mangoes enter the vision chamber by a roller conveyor system. During
movement, the mango is rotated around the roller axis and captured at different positions by a
camera to accurately predict its external features. The mangoes then move into the tray conveyor to
weigh and sort them based on the central processing signal. After that, in the central processing unit,
the mangoes’ weight and external features are combined to generate a data set which prepared for
the training in ML models. The unstructured data is converted to structured data, then it is put into
models of ML to predict the grades. Last but not least, the data transmission is carried out on the
server to ensure all processed data in the best.

(a) (b)
Figure 2.
Figure 2. The
Theprocess
process of weight
of weight system.
system. (a)load-cell
(a) The The load-cell in system;
in system; (b) flowchart
(b) flowchart of load-cellofsignal
load-cell signal
processing.
processing.
A system in Figure 1 is designed and developed for this study. The sorting system consists of
two main parts. The
The density first part
depends is the image
on weight processing
and volume system
which that implements
is a function conventional
of size. The image
change in weight
processing
and size ofusing a series
mangoes of algorithms
gradually that
increases convert
with unstructured
the mango’s data
quality. into
The structured
quality data to such
parameters extract
as
three external features such as length, width, and defect. The external features of mangoes can
be extracted by various methods such as image processing, ML, and deep learning. In this study,
the image processing is more efficient, because of its fast processing time without a large training
dataset. This method changes processing parameters, which are determined more difficult in both ML
and deep learning. The extracted(a) features of the captured image are combined (b) with weight to generate
a fullFigure
data set in order
2. The to of
process leverage machine
weight system. (a)learning models.
The load-cell The second
in system; part isofthe
(b) flowchart weighting
load-cell signal and
grading system to classify mango into G1 , G2 , and G3 .
processing.
In the sorting system, the mangoes enter the vision chamber by a roller conveyor system.
DuringThe density depends
movement, the mango on weight andaround
is rotated volumethewhich
rollerisaxis
a function of size.
and captured at The change
different in weight
positions by
aand size of
camera mangoes gradually
to accurately predict itsincreases
externalwith the mango’s
features. The mangoesquality. The
then quality
move intoparameters such as
the tray conveyor
to weigh and sort them based on the central processing signal. After that, in the central processing
unit, the mangoes’ weight and external features are combined to generate a data set which prepared
for the training in ML models. The unstructured data is converted to structured data, then it is put
into models of ML to predict the grades. Last but not least, the data transmission is carried out on the
server to ensure all processed data in the best.
Appl. Sci. 2020, 10, 5775 4 of 25

Appl.
Appl. Sci.
Sci. 2020,
The 10,
10, xx FOR
density
2020, PEER
PEER REVIEW
depends
FOR on weight and volume which is a function of size. The change in weight44 and
REVIEW of
of 25
25
size of mangoes gradually increases with the mango’s quality. The quality parameters such as size,
size,
size, shape,
shape, color, color,
shape, color, total
total soluble
total soluble soluble solids (TSS),
(TSS), acidity,
solids acidity,
solids (TSS), acidity,
pH, pH,
pH, physiological
physiological
physiological weight,
weight,
weight, juice, juice,
andpulp,
juice,
pulp, pulp, and
and
moisture
moisture
moisture
content content
arecontent
important are
are important
forimportant
evaluating thefor evaluating
for external
evaluating the external
the external
and internal and internal
andThey
qualities. internal qualities.
qualities. by
are determined They
They are
are
various
determined
determined which
parameters, by
by various
various parameters,
involveparameters, which
which involve
NDT techniques. involve
However,NDT
NDTthetechniques.
techniques. However,
However,
most important the
the most
parameter most important
important
is sweetness
parameter
parameter
from is
is sweetness
evaluating sweetness from
from evaluating
the internal evaluating
quality. The the internal
internal quality.
thesweetness quality.
of mangoThe sweetness
The depends
sweetnesson of mango
ofits
mango depends
depends
density. Mango on
on its
its
of
density.
density.
high densityMango
Mango has ofof high
thehigh density
density
better qualityhas the
hasinthe better
thebetter quality
range quality in
of moreinthanthe range
the 1.0.
range of
The more
ofmost than
moresort
than 1.0. The
1.0.mangoes
after most
The mosthave sort
sort
after
after mangoes
mangoes such
characteristics have
haveas characteristics
bright yellowsuch
characteristics skinas
such as bright
bright
and yellow
yellow In
sweetness. skin
skin and
and sweetness.
contrast, a mango In
sweetness. In contrast,
with a loweraadensity
contrast, mango
mango
with a
with 1.0
than lower
a lower density
density
is of poor than 1.0
thanit1.0
quality; is of poor
is ofbe
could poor quality;
sourquality; it
becauseitofcould
could be
lack be sour because
sour because
of nutrients of lack of
in theofmaturity nutrients
lack of nutrients in
in the
stage. Figure the3
maturity
maturity
shows stage.
relationship Figure
stage. Figure 3
between shows
3 shows relationship
relationship
sweetness between
between
and density. sweetness
sweetness
Therefore, weight and
and density.
density.
a factor Therefore,
Therefore,
to classify weight
quality alongaa
weight
factor to
to classify
factorsize
with classify
and defect. quality
quality along
along with
with size
size and
and defect.
defect.

Figure
Figure 3.
3. Relationship
Figure 3. Relationship between
between sweetness and density.
sweetness and density.
density.

The operating
The operating procedure
procedure of
procedure ofthe
of thesystem
the systemis
system isissummarized
summarized
summarized in Figure
inin Figure
Figure 4.
4. Input data
4. Input
Input is
is processed
datadata is processed
processed and
and
then
and ML
then is
ML applied
is appliedto classify
to mango
classify mango simultaneously
simultaneously byby the
the central
central processor.
processor.
then ML is applied to classify mango simultaneously by the central processor. The input data The
The input data
data
undergoes
undergoes several
several image
image processing
processing algorithms
algorithms to to achieve
achieve structured
structured data including length,
data including length, width, width,
defects,
defects, and
and weight,
weight, then
then saves
saves all
all this
this data
data to the server.
server. At the same time, the central processor
defects, and weight, then saves all this data to the server. At the same time, the central processor takes takes
data
data from
data from the
from the server
the server to
server to train
to train the
train the model
the model
modeland and then
andthen gives
thengives predictions
givespredictions whether
predictionswhether
whetherthethe mango
mangoisis
themango isGG
G111,,,G
G
G222,, or G333..
or G

Figure 4. Operating procedures of the system.

Figure
Figure 4.
4. Operating
Operating procedures
procedures of
of the
the system.
system.
In many references, supervised ML models were chosen to apply because these are simple and
In
In many
many references,
references, supervised
supervised ML ML models were chosen
chosen to apply because these are
are simple and
common models in the classification thatmodels wereevaluated
have been toto
apply becausein
be effective these
a number simple and
of other
common models in the classification that have been evaluated to be effective in a number
common models in the classification that have been evaluated to be effective in a number of other of other
studies
studies about
about grading
grading fruits
fruits with
with artificial
artificial intelligence.
intelligence. These
These models
models are are simple
simple and
and require
require fewfew
operating
operating resources
resources thus
thus bring
bring an
an advantage
advantage in in processing
processing time
time for
for the
the system.
system. ML ML is
is used
used toto train
train
machines
machines to
to handle
handle the
the data
data more
more efficiently
efficiently based
based on
on algorithms
algorithms and
and statistical
statistical models.
models. InIn this
this study,
study,
Appl. Sci. 2020, 10, 5775 5 of 25

studies about grading fruits with artificial intelligence. These models are simple and require few
operating resources thus bring an advantage in processing time for the system. ML is used to train
machines to handle the data more efficiently based on algorithms and statistical models. In this study,
applying ML is implemented to grade and sort the mango by learning from data. A supervised
ML algorithm is used with external assistance. The input data set is divided into testing, validation,
and training data. Supervised ML is the algorithm that creates a function that maps input data to
the desired outputs, which are appropriate for the classification problems. The supervised machine
learning algorithm learns from training datasets for classifying the mangoes into different groups
based on desired standards. All algorithms learn some kind of sample from the training dataset and
apply them to the testing data for classification. The four common models are discussed include RF,
LDA, SVM, and KNN. The image process of sorting mango using ML is described in the schematic
of Figure 3. The software of all the experiments was carried out on Python 3.6 (Python Software
Foundation, Fredericksburg, VA, USA).

3. Data Preparation for Grading Process

The accuracy of the training model depends on the diversity of data. Therefore, mango data was
collected and measured in several harvesting seasons in many orchards. The features are measured
many times. The number of measurements is n, the average value A of feature is shown in Equation (1).
The ith absolute error ∆Ai at each measurement is given Equation (2). And the average absolute error
∆A is determined in Equation (3).
n
P
Ai
i=1
A= (1)
n
∆Ai = A − Ai (2)
n
P
Ai
i=1
∆A = (3)
n
where,∆A means the random error. The systematic error ∆A0 was caused by the measuring instrument
and the accidental error ∆A, so the absolute error is ∆A = ∆A + ∆A0 . So, the relative error δA given
by (4).
∆A
δA 100% (4)
A
Every measurement has errors. Therefore, the accumulated error B will be generated according to
the Equation (5).
B = F ( xi ) , 0 < i ≤ n (5)

where i is the index number of variable in the function F. The error ∆B is shown in Equation (6).
n
X
∆B = ∆xi (6)
i=1

All samples were measured and recorded. We should find the errors in each mango and determine
absolute errors of their features.

4. Image Processing
In this part, the vision machine is applied to the analysis of visible imaging. This process consists
of 3 steps shown in Figure 5. In the first step, the images are acquired through the roller conveyor
system inside the image processing chamber which is sealed and lighted. In the second step, captured
images are processed by multiple algorithms such as increase fps (frames per second), image noise
filter, edge detection, and boundary tracking. In the final step, length, width, and defect are extracted
Appl. Sci. 2020, 10, x FOR PEER REVIEW 6 of 25
Appl. Sci. 2020, 10, 5775 6 of 25
images are processed by multiple algorithms such as increase fps (frames per second), image noise
filter, edge detection, and boundary tracking. In the final step, length, width, and defect are extracted
to generate a dataset. A A reference
reference [5]
[5] described
described a threshold method. The The digital
digital images
images are
are converted
converted
into binary images, then it is is processed
processed via
via aa sequence
sequence ofof morphological
morphological image processing. Each
Each frame
frame
is handled through
through different
differentalgorithms
algorithmsincluding
includingfiltered
filterednoise,
noise,edge
edgedetection,
detection,and
andboundary
boundarytrace to
trace
detect
to objects
detect in the
objects in images [12]. [12].
the images The structure of the hardware
The structure and the and
of the hardware vision
thechamber are designed
vision chamber are
based on based
designed the required productivity
on the required of the system,
productivity of the therefore the flowthe
system, therefore of moving mango is
flow of moving treated
mango is
continuously
treated during its
continuously movement.
during its movement.

Figure 5. The
The framework
framework of
of Image
Image Processing.
Processing.

During the movement

During movement of of mango
mango inside
inside the
the vision
vision chamber,
chamber, the the image
image of of the mango
mango is sent to a
central processor
central processor consecutively.
consecutively.TheThesorting
sortingsystem
systemshould should handle
handle continuously
continuously forfor a certain
a certain timetime
so
so mango’s
mango’s images
images areare acquired
acquired in in real-time.
real-time. The The frame
frame rateisisa afactor
rate factorthat
thataffects
affects the
the accuracy
accuracy of
processing. The The accuracy
accuracy is
is proportional
proportional to the increase in frame rate. Therefore, Therefore, using algorithms to
suitable choice.
increase frame rate is a suitable choice. Besides,
Besides, the
the defect
defect is
isdetected
detectedfrom fromcaptured
captured images,
images, therefore
therefore
depth-aware video frame interpolation [13] is is used
used to to ensure
ensure the thesmall
smallerror
errorestimation.
estimation.The Thenewnewithith
frame (fi) is created from two two adjacent
adjacent frames
frames(f(fi−1
i−1)) and (fi+1
i+1),),the
thevalue
valueofofframe
frameraterateincreases
increases toto more
more
double. The new frame is synthesized
synthesized from
from22successive
successiveframes frames(f(fi−1 i−1) and (fi+1
i+1))by
byarbitrary-time
arbitrary-time flowflow
interpolation [14]. The intermediate frame (fi) isi relied on visual flow from image f (i −1) →((i+1)
interpolation [14]. The intermediate frame (f ) is relied on visual flow from image f and
i−1)→ 1) and
(i+image
image f(i+1)→(i−1) in Equations (7) and (8).
f (i+1)→(i −1) in Equations (7) and (8).
_ 
f fi→i+1 == ((11 −
i→i+1 i)22 ff(i−1)→(i+1)−−i(1
− i) i(1−−i)fi)f(i+1)→(i−1) (7)
(7)
(i −1) →(i +1) (i +1) →(i −1)
_
f  = −i(1 − i)f )→(i+1) +2i2 (1 − i)f(i+1)→(i−1)
i→i−1 (8)
fi→i −1 = −i(1 − i)f((ii−1
−1) →(i +1) +i (1 − i)f (i +1) →(i −1) (8)
A Gaussian filter [15] is used to solve noised image, then the boundaries of the objects are detected.
A Gaussian
The Kernel matrixfilter
slides[15] is used
across to solve
each row noised image,
and multiply by areathen
of thethe
image. Let µ, σ of
boundaries be the objects
mean, are
variance
detected. The
of Gaussian Kernel matrix
distribution, slides across
respectively. each
x and row
y are twoand multiply
variables by area of(9)the
of Equation image.
and Let μ,2 σ
A = 1/2πσ . be
mean, variance of Gaussian distribution, respectively. x and y are two variables of Equation (9) and
2
A = 1 2 πσ 2 . −(x−µx )2 −( y−µ y )
+
2σ2 2σ2y
G(x, y) = Ae x (9)
2
− ( x − μ x )2 − ( y − μ y )
+
After filtering the frames, the next step is 2
to reduceσ 2
2 σ 2
the number
of dimensions of the image (9)to
G ( x, y ) = Ae x y

detect mango position. The RGB images are converted to appropriate binary image depending on the
colorAfter filtering
values. the frames,
Conversion from the next RGB
original step is
toto reduce the
greyscale usesnumber of dimensions
NTSC method of the
[16]. Next, image to
conversion
detect mango position.
from grayscale to binaryThe RGB based
image imageson are converted
Otsu’s methodto appropriate binary image
[17]. The threshold depending
C separates oninto
pixels the
color values. Conversion
two classes. The weighted from original
sum RGB toofgreyscale
of variances the two uses
classes σ2w (Cmethod
NTSC [16]. Next,
) is defined conversion
by Equation (10).
from grayscale to are
The probabilities binary image based
q1 (Equation (11))onand
Otsu’s method [17].
q2 (Equation (12))The threshold
which are fromC separates
the I bins.pixels into
Variances
1 sum2of variances of the two classes σ w (C ) is defined by Equation (10). The
of the two classes are σ
two classes. The weighted
2 and σ2 mean. The means µ and µ of probabilities
1 2
2 q 1 and q 2 are calculated
using the Equations (13) and (14), respectively.
probabilities are q1 (Equation (11)) and q2 (Equation (12)) which are from the I bins. Variances of the
σ2w (t) = q1 (t)σ21 + q2 (t)σ22 (10)
Appl. Sci. 2020, 10, 5775 7 of 25

t
X
q1 (t) = P(i) (11)
i=1
I
X
q2 ( t ) = P(i) (12)
i=t+1

t
P
iP(i)
i=1
µ1 (t) = (13)
q1 (t)
I
P
iP(i)
i=t+1
µ2 (t) = (14)
q2 (t)
From Equations (11)–(14), the variance is calculated as Equations (15) and (16).

t
X P(i)
σ21 (t) = [i − µ1 (t)]2 (15)
q1 ( t )
i=1

I
X P(i)
σ22 (t) = [i − µ2 (t)]2 (16)
q2 ( t )
i=t+1

I(x,y) is the light intensity at a single pixel and INP(x,y) is the intensity of the pixel on the binary,
0 < x < Length and 0 < y < Width. The thresholding method is used to detect the color threshold.
Values exceeding the threshold value are set to 1 and vice versa values inside threshold value are set to
0. The input of the method is the gray image and the threshold value.
From the binary image, the edges of the object are highlighted, then the remaining thing is to
connect those points to form the boundary of the object. There are two methods of edge detection such
as algebraic and geometric methods, but algebraic methods give unstable results, so it is appropriate to
use geometric methods based on partial geometric differential equations. For a better understanding
of the algorithms in [18], a brief description is given. A model satisfies the maximum principle and
permits a rigorous mathematical analysis was used effectively. The algorithm to find the contour of
objects is done by two methods of mathematics and geometry. The boundary of the object Eint are
detected and interpolated into curves according to the Equation (17).

Z1
2 2
Eint = (α v0 (t) + β v00 (t) )dt (17)
0

where v(t) = (x(s), y(s)), s ∈ (0, 1), α > 0 and β > 0 are the factors that affect the elasticity and rigidity
coefficients of curve.
The edges Eext are detected based on Equation (18) with maximum curve possible ∇I(v(t)).
Combining both of the above formulas, edge detection, and boundary tracking E(v,α,β,λ) are shown
through Equation (19).
Z1
Eext = −λ ∇I (v(t)) dt (18)
0

Z1 Z1
0 2 2
E(v, α, β, λ) = (α v (t) + β v (t) )dt − λ 00
∇I (v(t)) dt (19)
0 0
Appl. Sci. 2020, 10, 5775 8 of 25

In the above section, the image processing methods are described to extract mango size easily and
quickly through a series of effective algorithms.

5. Extracting Mango External Features from Captured Images

The images were segmented with a level of 0 for the pixel area of the mango and 1 for the
remaining pixel area in each frame. The next task is to calculate the area of the pixels according to
the actual size. This is the step that greatly affects the accuracy of the process. With binary images,
the pixel area is used to estimate actual size by Equation (20).

L = KAboundary (20)

Length L is the length to be estimated, Aboundary is the number of pixels and K is a ratio constant
between size in pixel and the actual size. Moving objects make the distance from the camera to the
objects change, therefore the proportional constants are also changed. In other words, the rate of
constant changes with the distance from the camera to the object. Therefore, the K ratio factor should be
appropriately estimated. With the same length at each focal distance, the number of pixels representing
that length varies. Therefore, the closer to the camera is more pixels that represent for that length.
That means the area of each pixel will decrease as the distance decreases. To determine the size of the
mango from a binary image whose coefficient of K changes due to the motion of the mango, the scale
factor from real data is estimated K ≈ F(Length). This is a linear function because as the number of
pixels increases, the length also increases. So linear planning is an appropriate option. The length is L,
the number of pixels in the image is A, and the values are considered on n images. The average length
L is given by Equation (21).
n
1X
L= Li (21)
n
i=1

From Equation (19), the K̂ ratio factor can be found in Equation (22).
n
P
Ai Li − nAL
i=1
K̂ = n
(22)
2
A2i
P
− nA
i=1

The length L̂ = K̂Aboundary is estimated by K̂ in Equation (20). The coefficient of determination is

defined in Equation (23).
P 2 2
ei
P
2 (L − L̂)
r = 1− = (23)
P 2 ( P Li ) 2 P 2 (P Li )2
Li − n Li − n
The error of K is given in Equation (24).
n
1X 2
ε= (Li − (K̂A)) (24)
n
i=1

The defect of the mango is the damage on the surface caused by insects or collisions during its
growth that can be scars, dark, spots, etc. The defect areas are detected by the boundaries of the
rectangle (Figure 6). All defects are accumulated on the whole surface and then give the final defect
level of each mango. Therefore, the areas of its disability should first be detected and zoned to be
detected effectively based on the specific areas of the binary image. The disabled areas are quite small,
so they should be covered by the rectangle, so the area of the disability is the area of those rectangles.
The mango defects are identified by the total pixel’s area of the zero value inside the mango boundary
in the binary image. Because the pixel size ratio is K, the area of each pixel is K2 . Where Adefect is the
 Appl. Sci. 2020, 10, x FOR PEER REVIEW 9 of 25

detected effectively based on the specific areas of the binary image. The disabled areas are quite small,
so they should be covered by the rectangle, so the area of the disability is the area of those rectangles.
TheSci.
Appl. mango defects
2020, 10, 5775 are identified by the total pixel’s area of the zero value inside the mango boundary 9 of 25
Appl. Sci. binary
in the 2020, 10,image.
x FOR PEER REVIEW
Because the pixel size ratio is K, the area of each pixel is K2. Where Adefect is 9the
of 25
 ) is estimated defect; (∆de) is error of defect depending on the error (ε)
number of defect pixels, ˆ( de
detectedofeffectively
number based
defect pixels, onisthe
(de) specific defect;
estimated areas of(∆de)
the binary
is errorimage.
of defectThedepending
disabled areas areerror
on the quite(ε)small,
of K.
of K. We can calculate the actual defect by Equation (25).
so they
We should bethe
can calculate covered
actualby the rectangle,
defect by Equationso the
(25).area of the disability is the area of those rectangles.
total
2 + 2ε K ) Adefect
The mango defects are identified de by =
thede + Δpixel’s
de = ( Karea
2 of the zero value inside the mango boundary (25)
in the binary image. Because the de pixel ˆ + ∆de
= desize ratio=is(KK, + 2εK
the )Adeoff ect
area each pixel is K2. Where Adefect is(25)
the
number of section,
defect pixels,  ) is estimated defect; (∆de) is error of defect depending on the error (ε)
( de
In this the actual dimensions of mango are estimated through the implemented algorithms.
of K. We
Based on can calculate
a series the actual the
of calculation, defect
sizeby
inEquation (25).
pixel is determined to the actual size with an acceptable
 + Δdeon
error. The estimation is calibrated depending
de = de
the hardware of the machine. Besides, the size of
= ( K 2 + 2ε K ) Adefect (25)
defects on the mango’s surface is calculated based on summing defective areas.

Figure 6. Defects of mango’s surface.

In this section, the actual dimensions of mango are estimated through the implemented
algorithms. Based on a series of calculation, the size in pixel is determined to the actual size with
an acceptable error. The estimation is calibrated depending on the hardware of the machine.
Besides, the size of defects on the mango’s surface is calculated based on summing defective
Figure 6.
Figure 6. Defects
Defects of
of mango’s
mango’s surface.
surface.
areas.
6. Estimating Volume
In this section, theand Density
actual dimensions of mango are estimated through the implemented
6. Estimating Volume and Density
algorithms. Based on
The estimation ofavolume
series ofandcalculation, the sizeon
density depends in the
pixel is determined
shape to the
of the mango, actual
which size by
varies withregion
an The estimation
acceptable error. of volume
The estimationand isdensity depends
calibrated on the shape
depending on theofhardware
the mango, of which
the varies by
machine.
and country. Therefore, the shape is extracted before estimating its volume and density. The harvested
region and
Besides, the country.
size Therefore,
of defects the mango’s
shape is extracted isbefore estimating
basedits volume and density. The
mangoes from orchards are on the
shown in Figuresurface
7. These calculated
mango samples on
are summing
measured defective
by the vision
harvested
areas. mangoes from orchards are shown in Figure 7. These mango samples are measured by the
machine and then weighted to generate the data. The prepared data for training models are structured
vision machine and then weighted to generate the data. The prepared data for training models are
data converted from image from and weight signal. Thesignal.
estimation of volumeofand densityandconsists
densityof
6.structured
Estimating data converted
Volume and Density image and weight The estimation volume
3 consists
steps. Inofthe first step,
3 steps. In the length, width,
first step, and defect
length, width,areandextracted
defect arefrom the captured
extracted from the image by a image
captured camera.
Inby
the second step, the length, width, and defect are combined with the weight to create
a camera. In the second step, the length, width, and defect are combined with the weight to createby
The estimation of volume and density depends on the shape of the mango, the
which completed
varies
dataset.
region Finally,
and
the completed the volume
country. Therefore,
dataset. andthe
Finally, density
the shapeare
volume predicted
isand densityapproximately
extracted before based
estimating
are predicted on length
its volume
approximately andand
based width
density.
on and
The
length
then
andits
harvested errors
width are then
mangoes
and evaluated.
from The are
its orchards
errors schematic
are shown of in
evaluated. the preprocessing
Figure
The 7. Theseof
schematic procedure
mango for predicting
samples
the preprocessing are measuredthe volume
procedure byfor
the
and
visiondensity is
machine shown
and in
then Figure
weighted8. to generate
predicting the volume and density is shown in Figure 8. the data. The prepared data for training models are
structured data converted from image and weight signal. The estimation of volume and density
consists of 3 steps. In the first step, length, width, and defect are extracted from the captured image
by a camera. In the second step, the length, width, and defect are combined with the weight to create
the completed dataset. Finally, the volume and density are predicted approximately based on length
and width and then its errors are evaluated. The schematic of the preprocessing procedure for
predicting the volume and density is shown in Figure 8.

Figure7.
Figure 7. Images
Images taken
taken from
from mangoes
mangoes of
ofone
onelocal
localspecies.
species.

Figure 7. Images taken from mangoes of one local species.

Appl. Sci. 2020, 10, 5775 10 of 25
Appl. Sci. 2020, 10, x FOR PEER REVIEW 10 of 25

Figure
Figure8.8.The
Theestimation
estimationprocess
processof
ofvolume
volumeand
anddensity.
density.

The internal
The internal quality
quality of
of mango
mango is is high
high ifif its
its density
density is is higher
higher than
than the
the average
average level.
level. Farmers
Farmers can can
use their
use their experience
experience to to evaluate
evaluate the the internal
internal quality
quality by by feeling
feeling the
the mango
mango in in their
their hands.
hands. The
The mango
mango
volume (V)
volume (V)hashas many
many deterministic
deterministicmethodsmethodssuch suchas as modeling,
modeling, statistical
statisticalanalysis
analysisbased
basedon on size
size or
or
weight. In
weight. Inthis
thisstudy,
study, the
the volume
volume of ofmangoes
mangoesisiscalculated
calculatedbasedbasedon ontwo-dimensional
two-dimensionalimages imagesbecause
because
this method
this method not not only
only requires
requires little
little resources
resources but but also
also has
has fast
fast processing
processing time.
time. Since
Since the
the mango
mango has has
a complex shape that helps it rotate on the roller, the image processing gives images of a mango in in
a complex shape that helps it rotate on the roller, the image processing gives images of a mango a
a random
random orientation.
orientation. In In several
several references,
references, thereare
there arethree
threevariables
variablestotodetermine
determinethe the volume
volume of of
mango. However,
mango. However, other otherstudies
studies [19,20] have
[19,20] have shown that that
shown widthwidth
(wi) and(wi)length
and (le) are related,
length (le) aretherefore,
related,
the volume
therefore, thecan be determined
volume by wi andby
can be determined le instead
wi and leofinstead
three variables. The processed
of three variables. images allow
The processed imagesus
to extract
allow us to theextract
valuestheof the mango
values length
of the and width.
mango length Inand thewidth.
extraction
In theof features,
extractiontheoforientations
features, the of
mangoes areofrandom
orientations mangoes arenrandom
with positions presented
with n positions in Figure
presented9a. The mango9a.
in Figure is detected
The mango by is
a rectangle
detected
covering
by a rectanglethemcovering
based on image
them basedprocessing
on imagealgorithms. During the During
processing algorithms. sampling theprocess,
sampling le and wi are
process, le
and wi arenextracted
extracted times at nnpositions.
times at nThe processThe
positions. of detecting
process ofmangoes
detecting and andleextracting
extracting
mangoes and wi arelegiven
and
in are
wi Figure
given 9b.inInFigure
Figure9b.9b,Intwo features
Figure are features
9b, two determined into n value into
are determined pairsn (wi,le) pairs ith
value from mango
(wi,le) fromatithn
positions.
mango
Appl. Sci.at nExperimental
2020, positions. results
Experimental
10, x FOR PEER REVIEW show that the
results show method is effective
that the method in is mango
effectiveininVietnam.
mango in Vietnam. 11 of 25
The collected data from images shows the volume depending on two variables as length (le),
width (wi). An actual process to measure reality mango is carried out with variables like le, wi, and V
in m samples of mango. The task is to predict the volume with the length and width. So, for the
regression method, le and wi are independent variables and Vˆ is a dependent variable that is
calculated by Equation (26).

Vˆ = b0 +b1le+b2wi (26)

When predicting volume, there is always an error ε. The coefficients of the variables are b0 , b1, b2 .
The actual volume V is calculated in Equation (27).

V = b0 + b1le + b2 wi + ε (27)
(a) (b)
Figure 9. Extracting
Extracting the length and width based on directions of mango. (a) The mango is detected by
a rectangle
rectanglecovering
coveringthem
them based
based on image
on image processing
processing algorithms,
algorithms, (b) the(b) the process
process of detecting
of detecting mangoes
mangoes and extracting length
and extracting length and width. and width.

Theevaluate
To collectedthe
data from images
accuracy of this shows the volume
regression depending
expression, we need ontotwo variables
make the sumas of
length (le),
squared
width (wi).asAn
residuals actual
small process with
as possible to measure
the sum reality mangoresiduals
of squared is carried∆V
out with variables
determined like
by the le, wi, and
Equation V
(28).
in m samples of mango. The n
task isn
to predict the volume
n
with the length and width. So, for the
ΔV =  ε i2 =  (Vi - Vˆi )2 =  (Vi - ( b0 + b1lei + b2 wii ) )
2
(28)
i=1 i=1 i=1

The density which varies between mangoes is determined from volume and weight. The internal
quality of mango is very important to grade the quality of the mango but has not been considered in
previous studies of mango classification [2–9]. The density D of a mango which is given in Equation
(29) is estimated by the weight (we) that get from the load-cell and the volume is predicted by le and
Appl. Sci. 2020, 10, 5775 11 of 25

regression method, le and wi are independent variables and V̂ is a dependent variable that is calculated
by Equation (26).
V̂ = b0 + b1 le + b2 wi (26)

When predicting volume, there is always an error ε. The coefficients of the variables are b0 , b1 , b2 .
The actual volume V is calculated in Equation (27).

V = b0 + b1 le + b2 wi + ε (27)

To evaluate the accuracy of this regression expression, we need to make the sum of squared
residuals as small as possible with the sum of squared residuals ∆V determined by the Equation (28).
n n n
2
X X X
∆V = ε2i = (Vi − V̂i ) = (Vi − (b0 + b1 lei + b2 wii ))2 (28)
i=1 i=1 i=1

The density which varies between mangoes is determined from volume and weight. The internal
quality of mango is very important to grade the quality of the mango but has not been considered in
previous studies of mango classification [2–9]. The density D of a mango which is given in Equation
(29) is estimated by the weight (we) that get from the load-cell and the volume is predicted by le and wi.
we
D = ∆D + (29)
b0 + b1 le + b2 wi

The density function is calculated based on we and V that have an error in the estimation. Therefore,
the error of the density function error is a cumulative error of we and V. Therefore, the error of the
density function is determined within a range of tolerances. If the cumulative error is too large, it is a
bad estimate in this case. From that point of view, the weight error is ∆we, and the volumetric error is
∆V. The cumulative error ∆D is determined in Equation (30).
n
1X
∆D = ∆we + ∆V = (∆wei + ∆Vi ) (30)
n
i=1

The cumulative error ∆D is compensated for errors of calculating and estimating the density to
ensure the least error in the estimates. This section ends, the volume and density that significantly
affect the quality of mangoes have been estimated by a linear regression method.

7. Using Machine Learning for Grading Mangoes

The internal quality of mango is a nonlinear function of features such as length, width, defect,
and weight. The quality of mango is graded based on the local standard (VietGAP) or international
standards (Global GAP), which are implemented easily to classify according to each feature, but it is
difficult to grade the fruits with combination of different features. The relationship of features is found
in several methods such as regression, statistical, and machine learning. In this paper, ML is proposed
to grade the internal quality of mango, because of its strong ability in calculating and analyzing
repeated problems. The papers [2–12] looked particularly closely at the application of supervised ML
techniques to mango classification. Hence, in this study, four supervised learning algorithms such as
SVM, LDA, KNN, and RF are proposed for grading and sorting mango. The trained models are based
on the sampling mango surveys in many orchards in Vietnam. In detail, the samples of mango are
randomly selected, and then their types are determined (Figure 10) by evaluating standards accurately.
(de). This manual grading is carried out by agricultural experts from Vietnam. The labeled types
including G1, G2, and G3 of mangoes are measured by D, V, and de from four extracted features de, we,
wi, and le. Instead of estimating density and volume to classify mango type, in automatic classification,
four models namely RF, LDA, KNN, and SVM are trained and evaluated by the labeled dataset
obtained from
Appl. Sci. 2020, 10, the
5775manual grading. The hardware and software of the sorting system were presented
12 of 25
in Section 2 and the grading standard is shown in Table 1.

Figure 10. The

The process
process of
of classification
classification system using machine learning.

Dataset is generated by manually grading

Table mango
1. Standard based
type on density (D), volume (V), and defect
mango.
(de). This manual grading is carried out by agricultural experts from Vietnam.2 The labeled types
Types Volume Range (mL) Length Range (cm) Width Range (cm) Defect (cm ) Density (g/mL)
including G1 , G2 , and G3 of mangoes are measured by D, V, and de from four extracted features de, we,
G3 250–400 10–12 6–8 >5 <1
wi, and
G2 le. Instead of estimating density
401–650 and volume to classify
12.1–14 8–9 mango type, in
3–5automatic classification,
1–1.1
fourG1
models namely RF, LDA, KNN, and
651–800 SVM are trained and
14.1–16 evaluated by the
9–10 labeled dataset
0–3 obtained
>1.1
from the manual grading. The hardware and software of the sorting system were presented in Section 2
and the
Thegrading
datasetstandard is shown
is divided in Table
into three parts1.as training, validating, and testing data. The ten-fold
Nested CV (NCV) [21] was used to separate data. In the outer layer, 10% of the original data was
separated for testing data to determine Table Standard type of
the1. performance mango.
models. The rest of the data was used to
develop a model. The rest
Volume Range of the 90% of the
Length Range original data was
Width Range used in the inner layer for the tuning of
Types Such data are separated into training data for the modelDefect
parameters. (cm2 )a prediction
to provide Density (g/mL)
or quality
(mL) (cm) (cm)
assessment,
G3 the validation
250–400 data are to evaluate the model’s
10–12 6–8 accuracy and >5 select the best<1 parameters
of the G2
model based on the given 12.1–14
401–650 output objectively. 8–9This process 3–5 not only provides 1–1.1 the best
G1
performance 651–800 but also14.1–16
for algorithms controls overfitting. 9–10 >1.1
0–3 of mango is implemented
The classification
by four steps including normalizing data, eliminating outliers, fitting models, model evaluation.
During
Thethe data collection
dataset is dividedprocess, there
into three are some
parts data points
as training, outsideand
validating, the testing
distribution
data. ofThe
a statistical
ten-fold
dataset called outliers, these points cause a significant error in the classification
Nested CV (NCV) [21] was used to separate data. In the outer layer, 10% of the original data was model. Thus, the
dataset needs to be eliminated outliers to create a better new dataset. The outliers
separated for testing data to determine the performance of models. The rest of the data was used tocan be detected and
removed
develop a based
model.on therest
The normalization
of the 90% ofmethod suchdata
the original as feature
was used scaling, min-max,
in the inner layerand
for thez-score,
tuningetc.of
Depending on the purpose, one of the above methods can be used most flexible. The
parameters. Such data are separated into training data for the model to provide a prediction or qualitybest method for
mango data the
assessment, is the z-score [22]
validation databecause the normalizations
are to evaluate the model’sare based on
accuracy theselect
and meantheand variance
best of each
parameters of
variable.
the modelThe data
based onpoints are output
the given optimized in the distribution
objectively. This processofnot
theonly
variables.
providesBecause of performance
the best the features
are on significantly
for algorithms different
but also ranges,
controls the feature
overfitting. The with a larger scale
classification dominates
of mango the predicted
is implemented results
by four of
steps
machine learning. By the z-score normalization, the effect of different features scales
including normalizing data, eliminating outliers, fitting models, model evaluation. During the data is avoided on
prediction. Using z-score
collection process, there arenormalization,
some data points four outside
features the
aredistribution
reshaped toofbea between
statistical−3dataset
and 3.called
Data
outliers, these points cause a significant error in the classification model. Thus, the dataset needs
to be eliminated outliers to create a better new dataset. The outliers can be detected and removed
based on the normalization method such as feature scaling, min-max, and z-score, etc. Depending on
the purpose, one of the above methods can be used most flexible. The best method for mango data
is the z-score [22] because the normalizations are based on the mean and variance of each variable.
The data points are optimized in the distribution of the variables. Because of the features are on
significantly different ranges, the feature with a larger scale dominates the predicted results of machine
learning. By the z-score normalization, the effect of different features scales is avoided on prediction.
Appl. Sci. 2020, 10, 5775 13 of 25

Appl. Sci. 2020, 10, x FOR PEER REVIEW 13 of 25

Using z-score normalization, four features are reshaped to be between −3 and 3. Data points outside
this range
points are eliminated
outside this rangefrom are data. The effectiveness
eliminated from data.ofThe dataeffectiveness
normalizationofand elimination
data of outliers
normalization and
to the performance of the four models is illustrated in the experiment section.
elimination of outliers to the performance of the four models is illustrated in the experiment section.
The supervised
The supervised ML ML models
models andand implementation
implementation platforms
platforms suitable
suitable for
for the
the prediction
prediction of
of types
types of
of
mango are determined by comparing prediction accuracies. Additionally, the
mango are determined by comparing prediction accuracies. Additionally, the optimized parameters optimized parameters
of each
of each model
modelto tofit
fitthe
thedata
dataofofmango
mangoare arediscussed.
discussed. Following
Following analysis
analysis of the
of the datadata to determine
to determine the
the relationship of variables, models are fitted into their parameters to achieve
relationship of variables, models are fitted into their parameters to achieve the most effective the most effective
performance. In
performance. In the
the previous
previous studies,
studies, several
several classification
classification models
models demonstrate
demonstrate the the efficiency
efficiency inin
grading mango as SVM [3,5,8], LDA [5,6,9], RF [10], and KNN [11]. Hence, in this
grading mango as SVM [3,5,8], LDA [5,6,9], RF [10], and KNN [11]. Hence, in this paper, SVM, LDA, paper, SVM, LDA, RF,
RF, KNNKNN
and and models are applied
models and compared
are applied to find the
and compared to most
find accurate
the most model in mango
accurate model classification.
in mango
classification. The framework of the training process is shown in Figure 11, which includes sixoutput,
The framework of the training process is shown in Figure 11, which includes six parts: Input, parts:
Random
Input, ForestRandom
output, (RF), K-Nearest Neighbors
Forest (RF), K-Nearest (KNN), Linear(KNN),
Neighbors Discriminant Analysis (LDA),
Linear Discriminant and Support
Analysis (LDA),
Vector
and Machine
Support (SVM).
Vector Machine (SVM).

Figure Framework
11. 11.
Figure Frameworkof training process
of training based
process on on
based models of Support
models Vector
of Support Machine
Vector (SVM),
Machine Linear
(SVM),
Discriminant Analysis (LDA),
Linear Discriminant Random
Analysis Forest
(LDA), (RF),Forest
Random and K-Nearest
(RF), and Neighbors
K-Nearest (KNN).
Neighbors (KNN).

Model RF:
Model RF: in RF model,
in RF model, training
training data data consists
consists of of kk subset
subset randomly
randomly selected
selected with
with replacement.
replacement.
Hence, the
Hence, the set
set of
of kk subset
subset is denotedBB== {b
is denoted {bii:: 00 << ii ≤≤k},
k},where
wherebbi iisisthe
theithithsubset.
subset. TheThepredicted
predicted label
label is
is
determined by majority vote method from k decision tree in RF.
determined by majority vote method from k decision tree in RF. In training process, RF model In training process, RF model separates
four features
separates fourinto nodesinto
features to lead
nodes theto final
lead decision
the final node. Low node.
decision correlation and reasonable
Low correlation and strength
reasonable of
the trees are two criteria to improve the performance of RF [23].
strength of the trees are two criteria to improve the performance of RF [23]. Consulting many Consulting many parameters of RF,
two criteria of
parameters canRF,betwo
achieved
criteriabycanthebe parameters
achieved by such theasparameters
the number of observations
such as the number that
ofare drawn for
observations
eachare
that treedrawn
(sample_size), splitting
for each tree criteria in the
(sample_size), nodescriteria
splitting (splitting_rule),
in the nodes the (splitting_rule),
number of trees the in the forest
number
(n_estimators), and the maximum depth of each tree (max_depth). In
of trees in the forest (n_estimators), and the maximum depth of each tree (max_depth). In sample_size, sample_size, decreased sampling
size produces
decreased more separate
sampling trees and
size produces more hence
separatea lower treescorrelation
and hence between
a lower thecorrelation
trees, which positively
between the
affectswhich
trees, the accuracy
positivelyof aggregated
affects theprediction
accuracy from trees. However,
of aggregated the precision
prediction from trees.of individual
However,trees the
decreases,of
precision as individual
the trainingtreesrequires fewer observations.
decreases, as the training Therefore,
requires Bootstrapping, which uses
fewer observations. random
Therefore,
samples with replacement
Bootstrapping, to control
which uses random the sample
samples with size of the trees.
replacement In splitting_rule,
to control the sample thissize is
ofone
the of the
trees.
main characterizing the RF. There are two rules, such as Gini impurity
In splitting_rule, this is one of the main characterizing the RF. There are two rules, such as Gini or Entropy, but Gini impurity
is the better
impurity option forbut
or Entropy, this analysis,
Gini impurity as Gini
is the canbetter
minimize option misclassification
for this analysis, andasfind
Ginithe largest
can class
minimize
while Entropy finds
misclassification and class
findgroups that account
the largest class while for Entropy
about 50% of data.
finds Hence, that
class groups the computing
account fortime Gini
of 50%
about
impurity is faster than Entropy. Each tree in the forest is created by Gini
of data. Hence, the computing time of Gini impurity is faster than Entropy. Each tree in the forest is impurity [24]. In the maximum
depth ofby
created each
Ginitree and the
impurity number
[24]. In theof trees in the
maximum forest,
depth increasing
of each tree andmax_deep
the number and ofn_estimators
trees in the lead to
forest,
deeper treesmax_deep
increasing and more trees, respectively thereby
and n_estimators lead tolonger deeper computation
trees and more time. Nevertheless, the prediction
trees, respectively thereby
longer computation time. Nevertheless, the prediction efficiency grows up with their increase. Hence,
the choice of max_deep and n_estimators can be seen as a trade-off between strength and the accuracy
of the trees.
Appl. Sci. 2020, 10, 5775 14 of 25

efficiency grows up with their increase. Hence, the choice of max_deep and n_estimators can be seen
as a trade-off between strength and the accuracy of the trees.
Model KNN: Let K be the number of nearest neighbors; the predicted label is aggregated from K by
Large Margin Nearest Neighbor algorithm (LMNN) [25]. In KNN, the number of neighbors is the core
deciding factor. The outcome is significantly influenced by the noise of a limited number of neighbors.
However, a large number of neighbors have expensive computations. Therefore, the number of
neighbors depends on kinds of data sets.
At the same time, KNN and RF classify three mango types, while SVM and LDA use the ‘One_vs_all’
method [26] only to grade a single classifier per class.
Model SVM: The output is classified based on three hyperplanes hp1 , hp2 , and hp3 . Because three
types of mango are classified using hyperplanes generated by various kernels, the kernel function [27]
is the main parameter of SVM. The kernel functions (rbf ) are proposed in this study. The parameter
C and γ influence the level penalty for misclassification and the complex boundary of separation,
respectively. For different data sets, these parameters are different. In this study, γ and C ranged from
0.3 to 1 and 70 to 100, respectively.
(θ ) (θ ) (θ )
Model LDA: three linear lines h1 1 , h2 2 , h3 3 are found to separate three types including G1 ,
G2 , G3 , . . . . The main factor of the LDA model is the number of returned features which reduce the
features of the initial data.
In this section, four supervised learning approaches on the theory and how to apply them to
the data in this paper were discussed. The advantages and disadvantages of each model are distinct.
The next experiment section will provide insight into the relevance of these models to the existing data
set. To evaluate the performance of models, we proposed the well-known and persuasive evaluation
metrics for classification (precision, sensitivity, F-measure (F1 score), and accuracy) [28].

8. Experiments and Discussions

These experiments used 4983 mango samples harvested from November to June to meet the
requirements of standards. They were measured accurately by the sorting system of the vision
machine and weight. Acquired data from grading work was important and necessary to predict
exact grades. The data was collected from the vision machine to get length, width, and area of the
defect. This data combined the weight of each mango measured based on load-cell. The parameters
of ML were determined to grade the mango quality employing input and output sets. The features
of the original data set were collected by image processing and weight. Besides, their labels were
classified by the manual classification system. Training and validation data from 90% of the original
data were separated by using the NCV method. Besides, testing data accounted for the remaining
10% of the original data. The ML models were implemented to grade the mango quality based on its
external features and weight. The models gave the results for 3 types of mangoes as good grade (G1 ),
medium grade (G2 ), or bad grade (G3 ). Besides, these models also showed the relations between the
input and output with different features. The models predicted the quality based on the qualification
standards automatically instead of the manual evaluation. The credibility of models was confirmed by
the empirical method using the confusion matrix to evaluate the performance of models. The accuracy
of the image processing part was evidenced by the experimental results in which actual and estimated
values are compared. The data was measured actually in with length, width, and area of defect by
Mitutoyo’s tools with an accuracy of 0.05 mm. The testing data checked the final solution in order
to confirm the actual predictive work. Besides, the weight was the subset of testing set measured by
electronic scales with an error of 0.01 g. Thereby, the volume (V) of the mango was determined by
the overflow method with 1000 mL glass jar with accuracy 0.4 mL. Each feature of mango such as
weight, width, and length was measured 10 times to calculate the expected value. From the above
sections, the models are recommended to predict types of mangoes. There are four models to apply
and implement such as RF, LDA, SVM, and KNN fitted by data.
Appl. Sci. 2020, 10, 5775 15 of 25

The mango samples were collected to generate data set applying to the machine learning models
considered in previous sections. Then, the accidental error was determined from the comparison
between the actual value and the estimated one. The measurement error was smaller, so we can
say that this data is reliable and considered as the prepared dataset for machine learning algorithms.
Then actual sizes were determined and compared with predicted values which are extracted by the
vision machine to give the bias. Besides, the grading and sorting the fruit were carried out according
to the standard applied largely. The characteristics of length, width, volume, defect, and density are
described in Table 1. The column of density is not the appearance in any standard for grading the
mango or other tropical fruits because their grades depend on the external features without density.
The density of mango cannot be determined indirectly, therefore the density should be predicted
by the volume and weight estimated by captured images and load-cell, respectively. In this study,
the mangoes are graded and classified into three groups: G1 , G2 , and G3 with the highest quality
being first grade (G1 ). The steps of the image processing were experimented as shown in Figure 12.
The Sci.
Appl. vision
2020, machine system
10, x FOR PEER took the data from the captured digital images of camera and used
REVIEW 15 ofthe
25
platform to determine the parameters of mango using specific functions. The mangoes entered the
vision
visionmachine
chambertoby give the bias.
a roller Besides,
conveyor the grading
system and were and captured
sorting the byfruit were carried
a camera. In each outRGB
according
image,
to the
the standard
mango’s applied
defect, largely.
length, The characteristics
and width were extractedofthrough
length, the
width, volume,
image defect,
conversion and density
stages: are
Calibration
described
image, HSV in image,
Table 1.and Thebinary
column of density
image. In Figureis not
13, the
The appearance
distributionsinofany standard
length, width,for grading
and volume thein
mango or other
the dataset havetropical
a shapefruits
similar because their grades
to the Gaussian dependBesides,
function. on the external
we observed features
that without
length and density.
width
haveThe density
a linear of mangoto
relationship cannot
volume be determined
(V). Hence, the indirectly,
volumetherefore
of mangothe density should
is predicted from its be length
predicted
and
by the volume and weight estimated by captured images and
width based on the prediction model of linear regression shown in Equation (31). load-cell, respectively. In this study,
the mangoes are graded and classified into three groups: G1, G2, and G3 with the highest quality being
first grade (G1). The steps of the image V =processing
−1088.2 + 4.1lewere+experimented
11.7wi as shown in Figure 12. The (31)
vision machine system took the data from the captured digital images of camera and used the
The to
platform estimated
determine sizes asparameters
the length and width
of mangofromusing
pixelsspecific
of the binary imageThe
functions. of mango
mangoes were compared
entered the
with the actual ones. Additionally, the estimated values of volume by
vision chamber by a roller conveyor system and were captured by a camera. In each RGB image,external features were different
the
from actual
mango’s volumes,
defect, length,which determined
and width by the overflow
were extracted throughmethod.
the image Thus, we utilized
conversion three
stages: measures
Calibration
to compare the performance of the estimation features. These measures
image, HSV image, and binary image. In Figure 13, The distributions of length, width, and volume (Table 2) are the mean
in the dataset have a shape similar to the Gaussian function. Besides, we observed that length error
absolute error (MAE), the root mean square error (RMSE) [29], and the mean absolute percentage and
(MAPE) [30]. According to Table 2, the results show good performance. The
width have a linear relationship to volume (V). Hence, the volume of mango is predicted from its MAPE with the maximum
value of
length and0.01283
widthand theon
based minimum value of
the prediction 0.00486
model are acceptable.
of linear regressionThe shownMAEinofEquation
length, width,
(31). defect,
volume, and density are 0.65371, 0.50863, 0.02954, 5.79612, and 0.01376, respectively. Five features
show that there are small error distributions V = −1088.2due to+ 4.1 le + 11.7widifferences between RMSE and (31)
insignificant MAE.

Figure
Figure12.
12.The
Theprocess
processofofextracting
extractingdimensions
dimensionsand
anddefects
defectsof
ofmangoes.
mangoes.

The estimated sizes as length and width from pixels of the binary image of mango were
compared with the actual ones. Additionally, the estimated values of volume by external features
were different from actual volumes, which determined by the overflow method. Thus, we utilized
three measures to compare the performance of the estimation features. These measures (Table 2) are
the mean absolute error (MAE), the root mean square error (RMSE) [29], and the mean absolute
Appl. Sci. 2020, 10, 5775 16 of 25
Appl. Sci. 2020, 10, x FOR PEER REVIEW 16 of 25

Figure 13.
Figure Thelinear
13. The linear relationship
relationship between
between length,
length, width,
width, and
and volume.
volume.

Table 2. The accuracy of estimated features.

The importance of data normalization and elimination of outliers (DNEO) in model accuracy
was mentioned in Section 7. of
Feature The impact of the
Mango MAE DNEO process RMSEon model MAPEperformance is therefore
presented in this section. Aslength
can be seen from 0.65371
Table 3, four features
0.67623 are in 0.00513
very different ranges which
are determined by min andwidth max values of variables.
0.50863 The ranges0.51343 of we, he, wi account for (112,784),
0.00486
defect
(15,300), (16,300), respectively 0.02954
while that of defect only from0.032215 0.00813
(0–29) and it is the lowest scale. Although
volume
de is important as a predictor, 5.79612
it intrinsically influences the5.95234
result less due0.01161
to its smaller value. After
density 0.01376 0.01465 0.01283
using the DNEO process, in Table 4, there is a significant decrease of 218 outliers in the original data.
In Figure 14a, the data points beyond the smallest, and the largest observation is considered outliers.
Hence,The importance
there ofoutliers
are a lot of data normalization
to remove in andthis elimination
data. Figure of 14boutliers (DNEO)
indicates inrange
that the model ofaccuracy
features
was
is mentionedtoinabout
normalized Section 7. The
(−1712, impact
1737). the DNEO
Theofeffect of dataprocess on model
normalization and performance
eliminationisoftherefore
outliers
presented in this section. As can be seen from
(DNEO) on the accuracy of the models is shown in Figure 15. Table 3, four features are in very different ranges
which are determined by min and max values of variables. The ranges of we, he, wi account for
(112,784), (15,300), (16,300), respectively
Table 2. Thewhile that of estimated
accuracy defect only from (0–29) and it is the lowest scale.
features.
Although de is important as a predictor, it intrinsically influences the result less due to its smaller value.
Feature of Mango MAE RMSE MAPE
After using the DNEO process, inlength Table 4, there0.65371
is a significant decrease of 218 outliers in the original
0.67623 0.00513
data. In Figure 14a, the data points beyond the
width smallest,
0.50863 and the0.00486
0.51343 largest observation is considered
outliers. Hence, there are a lot ofdefectoutliers to remove
0.02954 0.032215 0.00813 14b indicates that the range
in this data. Figure
of features is normalized to about (−1712, 1737).
volume The effect
5.79612 of data0.01161
5.95234 normalization and elimination of
outliers (DNEO) on the accuracydensity of the models0.01376
is shown0.01465
in Figure0.01283
15.

Basic statistical
Table 3. Basic
Table statistical information
information on
on the original data.

Weight
Weight Length
Length Width
Width Defect Defect
Mean 450.066963 131.422130 85.191636 5.211869
Mean 450.066963 131.422130 85.191636 5.211869
Std Std 89.887420
89.887420 25.330647
25.330647 27.233621
27.233621 4.323399 4.323399
Min Min 112.000000
112.000000 15.000000
15.000000 16.000000
16.0000000.060000 0.060000
25% 25% 401.167236
401.167236 114.900000
114.900000 74.470000
74.4700002.550000 2.550000
50% 50% 449.835599
449.835599 130.500000
130.500000 81.790000
81.7900004.650000 4.650000
75% 75% 499.956623
499.956623 145.900000
145.900000 88.650000
88.6500007.100000 7.100000
Max Max 784.000000
784.000000 298.000000 299.000000
298.000000 299.00000029.00000029.000000
 Appl. Sci. 2020, 10, x FOR PEER REVIEW 17 of 25
Appl. Sci. 2020, 10, x FOR PEER REVIEW 17 of 25
Appl. Sci. 2020, 10, 5775 17 of 25
Table 4. Basic statistical information on the data using data normalization and elimination of
Table 4. Basic statistical information on the data using data normalization and elimination of
outliers (DNEO).
outliers (DNEO).
Table 4. Basic statistical information on the data using
Weight data normalization
Length Width and elimination of outliers (DNEO).
Defect
Weight Length Width Defect
Mean −50.000984 0.000703 −0.000904 0.000150
Weight 0.000703
Mean −50.000984 Length
−0.000904 Width
0.000150 Defect
Std 1.000211 0.999763 0.999931 1.000004
Std 1.000211 0.999763 0.999931 1.000004
Mean Min−50.000984 0.000703
−1.712000 −1.739000 −0.000904
−1.787000 −1.731000 0.000150
Std Min 25% −1.712000
1.000211 −1.739000 −1.787000 0.999931
−1.731000
−0.872000 −0.860000 −0.861000 −0.801000 1.000004
0.999763
Min25% −0.872000
−1.712000 −0.860000 −0.861000 −1.787000
−1.739000 −0.801000 −1.731000
50% −0.020000 0.008000 0.025000 −0.054000
25%50% −0.020000
−0.872000 0.008000 0.025000 −0.861000
−0.860000 −0.054000 −0.801000
75% 0.866000 0.866000 0.846000 0.864000
50%75% 0.866000
−0.020000 0.866000 0.846000 0.025000
0.008000 0.864000 −0.054000
75%Max Max1.734000 1.734000
0.866000 1.737000
0.866000
1.737000
1.703000
1.703000 0.846000
1.712000 0.864000
1.712000
Max 1.734000 1.737000 1.703000 1.712000

(a) (a) (b) (b)

Figure
Figure 14.implemented
14. The The
Theimplemented process
process
implemented of dataof
process data
data normalization
ofnormalization and elimination
and elimination
normalization of outliers.
of outliers. (a) A
A graphical
(a) A graphical
graphical
rendition of the original data based on the box plot; (b) a graphical rendition of the data using DNEO
rendition of the original data based on the box plot; (b) a graphical rendition of the data using DNEO
based on the
based onbox
theplot.
box plot.

Figure Figure
15. The15. Theof
effect effect of thenormalization
the data data normalization
and theand the elimination
elimination of outliers
of outliers on theonperformance
the performance of of
models.
Figure 15. (a)
The RF without
effect of theDNEO,
data (b) RF with
normalization DNEO,
and (c)
the LDA without
elimination
models. (a) RF without DNEO, (b) RF with DNEO, (c) LDA without DNEO, (d) LDA with DNEO, of DNEO,
outliers (d)
on LDA
the with
performance DNEO, of
(e) KNN without DNEO, (f) KNN with DNEO, (g) SVM without DNEO, (h) SVM with DNEO.
(e)models. (a) RF without
KNN without DNEO, (f) DNEO, (b) RF
KNN with with DNEO,
DNEO, (g) SVM(c) LDA without
without DNEO, DNEO,
(h) SVM(d) LDA
with DNEO. with DNEO,
(e) KNN without DNEO, (f) KNN with DNEO, (g) SVM without DNEO, (h) SVM with DNEO.
The The performances
performances of models
of models are evaluated
are evaluated by using
by using the confusion
the confusion matrixmatrix
because because
it observesit observes
the the
relations The
relations performances
between
betweenthe classifierof models
outputs
the classifier are
and
outputs evaluated
the
andtrue by
trueusing
the ones. The the
ones. confusion
elements
The matrix
in the
elements the because
diagonal
in diagonal j) ij(i, iis= j) (ithe
(nij, i it=(n
observes is
relations
row row
identifier between
and jand
identifier the
is the classifier
j iscolumn outputs
identifier)
the column and the
are the
identifier) are true
elementsones. The elements
correctly
the elements in
classified,
correctly the diagonal
while the
classified, while (n
elements , i
the elements
ij = j) (i is
out row
of
outtheidentifier
of diagonal and
the diagonal arej misclassified.
is
arethe columnFouridentifier)
misclassified. models
Four areare
models the elements
built
are with correctly
built two
with two classified,
choices: Without
choices: while
DNEO
Without the and
DNEO elementsand
without
with ofDNEO.
DNEO. the diagonal
Figure areshows
15 shows
Figure 15 misclassified.
the effect Four
of the
the effect models
ofdata
the are builtand
normalization
data normalization with twoelimination
choices:
elimination
and Without
of outliers on DNEO
of outliers fouron four and
with
models DNEO.
relyingFigure
on the15 shows
confusionthe effect
matrix. of A
thesignificant
data normalization
increase inand eliminationisof
performance outliers
shown in on all four
four
According to Figure 15, the accuracy of the RF model is the highest, the SVM model is the lowest.
Additionally, the prediction for class G2 in all models is lower in the two other classes. After the
implementation of the DNEO, result has unqualified performance because of uncontrolled overfitting.
Additionally, models’ parameters have still not been identified optimally. Varma and Simon [21]
showed that overly optimistic performance estimates can be produced by using the same data to
Appl. Sci. 2020, 10, 5775 18 of 25
validate and train models. They also suggested the unbiased performance estimates given in nested
cross-validation (NCV). NCV method used to tune parameters with an internal NCV loop while an
external
models, if NCV
the was
DNEO used to calculate
is used. an estimate
In the original of the
dataset, theerror. Considering
accuracy the accuracy
of RF, LDA, KNN, andofSVM the models
models
by the size
account for of the training
41.31%, 44.00%,data to select
41.31%, the number
and 44.28%, of folds After
respectively. in theusing
NCVDNEO,
methodthe (Figure 16). The
corresponding
horizontal
performances axisofdenotes the percentage
these models considerablyof the
grow initial
up todata used86.56%,
91.50%, for training
86.20%,models. The vertical
and 81.90%. axis
According
denotes
to Figurethe 15,accuracy of theofmodels
the accuracy the RF that
modelmatch
is thethehighest,
horizontal axis values.
the SVM model In Figure
is the 16, when training
lowest.
aboutAdditionally,
80% of the original dataset,for
the prediction theclass
accuracy
G2 in of allmodels
modelsisisthe
lowerhighest.
in theTherefore,
two other in both the
classes. Afterinner
the
and outer loops of the NCV method, 10 folds were used to ensure that 80%
implementation of the DNEO, result has unqualified performance because of uncontrolled overfitting. of the data was used in
training models.
Additionally, models’ parameters have still not been identified optimally. Varma and Simon [21]
The performance
showed that overly optimistic of the models was determined
performance estimatesby cantesting data divided
be produced 10 times
by using in thedata
the same outerto
layer. The estimation of accuracy (Figure 17) was implemented 10 times corresponding
validate and train models. They also suggested the unbiased performance estimates given in nested to 10 folds in
the outer layer. The
cross-validation performances
(NCV). NCV method of four models
used to tune with adjustingwith
parameters theiran
parameters wereloop
internal NCV evaluated
while by an
validation and testing data. The horizontal axis denotes the evaluating
external NCV was used to calculate an estimate of the error. Considering the accuracy of the models by iteration of model
performance
the size of theand the vertical
training data to axis denotes
select the percentage
the number of folds in of
themodel performance.
NCV method The
(Figure 16).black
The line in the
horizontal
figure presents
axis denotes thethe training of
percentage score
the and the
initial redused
data line for
denotes the models.
training validationThescore of models
vertical with the
axis denotes the
best parameters. The training and validation scores were determined by validation
accuracy of the models that match the horizontal axis values. In Figure 16, when training about 80% of and testing data,
respectively. Finally,the
the original dataset, theaccuracy
gray area aroundisthe
of models thered line shows
highest. the standard
Therefore, in both thedeviation
inner andinouter
the outer
loops
layer performance of each fold.
of the NCV method, 10 folds were used to ensure that 80% of the data was used in training models.

Figure
Figure 16.
16. The
The accuracy
accuracy of
of models
models based
based on
on training
training set
set size.
size.

The model’s
The performance of the models
performances showwasthatdetermined by testing
parameter selection indata divided
the NCV 10 times
method in the outer
is important for
layer. The estimation
controlling overfitting.ofThe
accuracy (Figure
separation 17) initial
of the was implemented 10 times
data into training andcorresponding
testing data givesto 10 folds in
a smaller
the outer
effect. Thelayer. The performances
accuracy of RF, LDA, KNN, of four models
and SVM with adjusting
increase their parameters
significantly were evaluated
to 98.1%, 87.2%, by
94.8%, and
validation and testing data. The horizontal axis denotes the evaluating iteration of
92.7%, respectively. The RF model obtains the highest accuracy at 98.1% if (n_estimators) is rangemodel performance
and the
from 80 vertical axisthe
to 100 and denotes
nodesthearepercentage
expandedof model
until performance.
all leaves are pure.The black
The line in of
accuracy thethe
figure
SVMpresents
model
the training
using Kernelscore and(rbf)
function the red line
is up denotes
92.7% withthe
γ andvalidation
C rangedscore
fromof0.3
models with
to 1 and 70 the bestrespectively.
to 100, parameters.
The training
Moreover, and
the KNNvalidation
model hasscores wereaccuracy
a high determined by validation
of 94.8% and testing
in the number data, respectively.
of neighbors ranging fromFinally,
30
the60.
to gray area when
Finally, around the red the
reducing linenumber
shows the standardreturned
of features deviationto in outermodel
1, the LDA layer performance
allocates 89.6%of
each fold.
accuracy.
The model’s performances show that parameter selection in the NCV method is important for
controlling overfitting. The separation of the initial data into training and testing data gives a smaller
effect. The accuracy of RF, LDA, KNN, and SVM increase significantly to 98.1%, 87.2%, 94.8%, and 92.7%,
respectively. The RF model obtains the highest accuracy at 98.1% if (n_estimators) is range from 80 to
100 and the nodes are expanded until all leaves are pure. The accuracy of the SVM model using Kernel
function (rbf) is up 92.7% with γ and C ranged from 0.3 to 1 and 70 to 100, respectively. Moreover,
the KNN model has a high accuracy of 94.8% in the number of neighbors ranging from 30 to 60. Finally,
when reducing the number of features returned to 1, the LDA model allocates 89.6% accuracy.
distributing data on the boundary clusters obviously. The clusters in the middle of the data field are
complex to classify. Besides, the area of density (0,0.67) and (1.03,1.2) are obvious, it is difficult to
determine the boundary of the remaining area of density (0.67,1.03). Because the area of density is
small or large, it is easy to identify the type of mango. Meanwhile, the mango classification of the
region
Appl.
(0.75,1.03)
Sci. 2020, 10,10,
5775
is difficult to decide the types of mango, because these types depend on other
1919
ofof
2525
Appl. Sci. 2020, x FOR PEER REVIEW
grading factors as weight and volume.
The relationship of the features is the linear or nonlinear relationships shown in Figure 18.
Therefore, we should not apply the linear model to grade of mango. The data is divided into 3 parts
including training data, validation data, testing data. The samples divided into three parts with 3194
training data visualized in Figure 19, remaining data includes 771 validation, and 1035 testing. The
distributing data on the boundary clusters obviously. The clusters in the middle of the data field are
complex to classify. Besides, the area of density (0,0.67) and (1.03,1.2) are obvious, it is difficult to
determine the boundary of the remaining area of density (0.67,1.03). Because the area of density is
small or large, it is easy to identify the type of mango. Meanwhile, the mango classification of the
region (0.75,1.03) is difficult to decide the types of mango, because these types depend on other
grading factors as weight and volume.

Figure
Figure17.
17.The
Theevaluation
evaluationofofmodel
modelperformance
performanceusing
usingthe
theNCV
NCVmethod.
method.

The relationship of the features is the linear or nonlinear relationships shown in Figure 18.
Therefore, we should not apply the linear model to grade of mango. The data is divided into 3 parts
including training data, validation data, testing data. The samples divided into three parts with
3194 training data visualized in Figure 19, remaining data includes 771 validation, and 1035 testing.
The distributing data on the boundary clusters obviously. The clusters in the middle of the data field
are complex to classify. Besides, the area of density (0,0.67) and (1.03,1.2) are obvious, it is difficult
to determine the boundary of the remaining area of density (0.67,1.03). Because the area of density
is small or large, it is easy to identify the type of mango. Meanwhile, the mango classification of the
region (0.75,1.03) is difficult to decide the types of mango, because these types depend on other grading
factors as weight and Figure 17. The evaluation of model performance using the NCV method.
volume.

Figure 18. The relationship between the features of mangoes.

Figure18.18.The
Figure Therelationship
relationshipbetween
betweenthe
thefeatures
features
ofof mangoes.
mangoes.

In the experiments, we used four models for grading and classifying. The first model LDA
was applied to grade the mangoes and it gave a relative accuracy of 87.9% shown in Figure 20.
The mangoes were clustered well with the density’s areas (0,0.67), and (1.03,1.2), where the mangoes
are graded exactly. However, the error increased in the region of density (0.67,1.03), because the
types of mangoes G1 , G2 , and G3 have complex distribution in this density area. Thus, a significant
difficulty of classification was seen in the created boundary by linear lines of the LDA method. Besides,
many mangoes were classified inexactly in the area of density (0.67,1.03), which is the intersection
between quality types of mangoes.
 Appl. Sci. 2020, 10, 5775 20 of 25
Appl. Sci. 2020, 10, x FOR PEER REVIEW 20 of 25

Figure 19. Visualization of training data.

In the experiments, we used four models for grading and classifying. The first model LDA was
applied to grade the mangoes and it gave a relative accuracy of 87.9% shown in Figure 20. The
mangoes were clustered well with the density’s areas (0,0.67), and (1.03,1.2), where the mangoes are
graded exactly. However, the error increased in the region of density (0.67,1.03), because the types of
mangoes G1, G2, and G3 have complex distribution in this density area. Thus, a significant difficulty
of classification was seen in the created boundary by linear lines of the LDA method. Besides, many
mangoes were classified inexactly in the area of density (0.67,1.03), which is the intersection between
quality types of mangoes. Figure 19. Visualization of training data.
Figure 19. Visualization of training data.

In the experiments, we used four models for grading and classifying. The first model LDA was
applied to grade the mangoes and it gave a relative accuracy of 87.9% shown in Figure 20. The
mangoes were clustered well with the density’s areas (0,0.67), and (1.03,1.2), where the mangoes are
graded exactly. However, the error increased in the region of density (0.67,1.03), because the types of
mangoes G1, G2, and G3 have complex distribution in this density area. Thus, a significant difficulty
of classification was seen in the created boundary by linear lines of the LDA method. Besides, many
mangoes were classified inexactly in the area of density (0.67,1.03), which is the intersection between
quality types of mangoes.

Classification of
Figure 20. Classification
Figure of LDA
LDA model.
model.

The second
The second model
model SVM SVM is is similar
similar toto LDA:
LDA: The The clusters
clusters of of mangoes
mangoes have have boundaries
boundaries of of the
the
hyperplanes.The
hyperplanes. Theresults
resultsof of grade
grade based
based on SVM
on SVM havehave an accuracy
an accuracy of 92.7%.
of 92.7%. The accuracy
The accuracy of the
of the model
model SVM
SVM depends on the Kernel function. The types of kernel functions considered for this study includestudy
depends on the Kernel function. The types of kernel functions considered for this linear,
include linear,
polynomial, rbf, polynomial,
and sigmoid. rbf,
The sigmoid. Theindicate
andexperiments experiments
that rbfindicate
is the mostthat rbf is the Kernel
efficient most efficient
function.Kernel
The
function. The results show that the classification of SVM is better
results show that the classification of SVM is better than the LDA model (Figure 21). than the LDA model (Figure 21).
Similarly, grading
Similarly, gradingresults
resultsevaluation
evaluationin in field
field of of density
density (0,0.67),
(0,0.67), and and (1.03,1.2)
(1.03,1.2) is exact.
is exact. The
The classification
classification results results
in theindensity
the density (0.67,1.03)
(0.67,1.03) alsomore
also give give more
accurate accurate
resultsresults
than thethan the model
model of LDA,of
LDA, because this area is clustered by the hyperplanes, and it makes the classification
because this area is clustered by the hyperplanes, and it makes the classification more flexible. The more flexible.
The classification based on the Figure
SVM 20. Classification
model is reliable. ofHowever,
LDA model. thereareareinexact
inexactgrades
gradesfor
for classifying,
classifying,
classification based on the SVM model is reliable. However, there
because intersections
because intersections of of types
types blend
blend together.
together. Another
Another model
model cancan support
support to to solve
solve inexact
inexact classification
classification
The second model SVM is similar to LDA: The clusters of mangoes have boundaries of the
being KNN. It is an algorithm that works and is considered based on predicted
being KNN. It is an algorithm that works and is considered based on predicted points, which affect points, which affect the
the
hyperplanes. The results of grade based on SVM have an accuracy of 92.7%. The accuracy of the model
classificationresults
classification resultsandandaccuracy.
accuracy.The Theaccuracy
accuracyofofthe theprediction
prediction is is
thethe proportion
proportion inversely
inversely neighbors
neighbors of
SVM depends on the Kernel function. The types of kernel functions considered for this study include linear,
of predicted points. If the number of neighbors are 30, the KNN model gives grading results shown in
polynomial, rbf, and sigmoid. The experiments indicate that rbf is the most efficient Kernel function. The
Figure 22.
results show that the classification of SVM is better than the LDA model (Figure 21).
The classification in density area (0.67,1.03) had more improvement than the SVM model, but in
Similarly, grading results evaluation in field of density (0,0.67), and (1.03,1.2) is exact. The
the remaining area, the KNN model is less reliable than the SVM model, because the boundaries of
classification results in the density (0.67,1.03) also give more accurate results than the model of LDA,
areas are distinct. Therefore, KNN is suitable for problems having many intersections. Three methods
because this area is clustered by the hyperplanes, and it makes the classification more flexible. The
LDA, SVM, and KNN have their own advantages as well as a disadvantage in classifying mangoes
classification based on the SVM model is reliable. However, there are inexact grades for classifying,
or fruits.
because intersections of types blend together. Another model can support to solve inexact classification
being KNN. It is an algorithm that works and is considered based on predicted points, which affect the
classification results and accuracy. The accuracy of the prediction is the proportion inversely neighbors of
Appl. Sci.
Appl. Sci. 2020,
2020, 10,
10, xx FOR
FOR PEER
PEER REVIEW
REVIEW 21 of
21 of 25
25

predictedpoints.
predicted points.IfIfthe
thenumber
numberof
ofneighbors
neighborsare
are30,
30,the
theKNN
KNNmodel
modelgives
givesgrading
gradingresults
resultsshown
shownin
inFigure
Figure
Appl. Sci. 2020, 10, 5775 21 of 25
22.
22.

Figure 21. Classification

Figure Classification of SVM
SVM model.
Figure 21.
21. Classification of
of SVM model.
model.

Figure 22.
Figure 22. Classification of KNN
Classification of KNN model.
model.

The final model ofin

The classification
classification RFdensity
overcomeareathe disadvantages
(0.67,1.03) had more
more of previous
improvement models. than This
themodel has differing
SVM model,
model, but in
in
The in density area (0.67,1.03) had improvement than the SVM but
accuracy depending
the remaining
remaining area, on
area, the the
the KNN number
KNN model
model isof trees
is less in the
less reliable forest.
reliable thanthan the If the
the SVM number
SVM model, of trees
model, because increases
because the in number
the boundaries
boundaries of of
the
from
areas 80
areto 100 trees,
distinct. the accuracy
Therefore, KNN is the RF for
ofsuitable model is between
problems having 97many
to 98.1%. Therefore,
intersections. the methods
Three selected
areas are distinct. Therefore, KNN is suitable for problems having many intersections. Three methods
parameters
LDA, SVM,
LDA, SVM, and andRF
for
KNN
model
KNN have
have
is to ensure
their
their ownthe
own
stability and
advantages
advantages
training
as well
as well speed of the number
as aa disadvantage
as disadvantage of treesmangoes
in classifying
in classifying (Figure 23).
mangoes or or
The disadvantages
fruits. of the previous models are solved in the area of (0,12) explicitly and accurately
fruits.
according to the
The final
final rules.
model The
of RF accuracy the
RF overcome
overcome depends on the complex
disadvantages of previous boundaries
previous models.between
This model
modeltypes
hasof mango.
differing
The model of the disadvantages of models. This has differing
From experimental
accuracy depending performances
on the
the number
numberof four models
of trees
trees LDA,
in the
the SVM,
forest. KNN, and RF, ofthe RFincreases
model is in selected to
accuracy depending on of in forest. IfIf the
the number
number of trees
trees increases in number
number
grade
from 80 mangoes’
80 to
to 100 quality.
100 trees,
trees, the Figure
the accuracy 24
accuracy ofshows
of the the
the RF comparison
RF model
model is and
is between
between 97 evaluation
97 to of
to 98.1%. grading
98.1%. Therefore, results
Therefore, the based
the selectedon
selected
from
four models for
parameters to give
RF the bestismodel.
model to ensureBesides,
the Table 5 and
stability shows sensitivity,
training speed precision,
of the F1 score,
number of and
trees accuracy
(Figure
parameters for RF model is to ensure the stability and training speed of the number of trees (Figure
as well
23). The as the differences
Thedisadvantages
disadvantages of between
of the models
the previous
previous models for
modelsare grading
aresolved [28].
solved in All
in the models
thearea
areaof give
of (0,12) accuracies
(0,12) explicitly
explicitlyand of more
andaccuratelythan
accurately
23).
87.9%. The
according to best
to the model
the rules.
rules. Theis RF with
The accuracy an accuracy
accuracy depends
depends on of
on the 98.1%.
the complex RF model
complex boundaries performed
boundaries between best
between types with
types of average
of mango.
mango.
according
values of sensitivity, precision, F1 score, and accuracy of 98.1%, 98.0%, 98.0%, and 98.1%, respectively.
From experimental performances of four models LDA, SVM, KNN, and RF, the RF model is selected
to grade mangoes’ quality. Figure 24 shows the comparison and evaluation of grading results based
on four models to give the best model. Besides, Table 5 shows sensitivity, precision, F1 score, and
accuracy as well as the differences between models for grading [28]. All models give accuracies of
Appl. Sci. 2020, 10, 5775 22 of 25
more than 87.9%. The best model is RF with an accuracy of 98.1%. RF model performed best with
average values of sensitivity, precision, F1 score, and accuracy of 98.1%, 98.0%, 98.0%, and 98.1%,
respectively.
The sensitivity Theandsensitivity and
precision precision
values showvalues show of
that 98.1% that 98.1%
three of three
types (G1 , types
G2 , and(G1G, G
3 )2,were
and G 3) were
correctly
correctly
classifiedclassified as corresponding
as corresponding three typesthree
andtypes
98.0%and 98.0% correctly
correctly predictedpredicted
types to the types totalto predicted
the total
predicted
correspondingcorresponding types.the
types. Besides, Besides,
lowest the lowest performance
performance is LDA model is LDA
with model
averagewithvalues average values
of sensitivity,
of sensitivity,
precision, precision,
F1 score, and F1 score, and
accuracy accuracy of, 87.8%,
of, respectively, respectively,
87.9%,87.8%,
87.8%,87.9%, 87.8%,In
and 87.9%. and 87.9%. In
comparison,
comparison,
the KNN model the KNN model
results results
yielded yielded
average average
values values ofprecision,
of sensitivity, sensitivity,F1precision,
score, and F1accuracy
score, and of,
accuracy of, respectively,
respectively, 94.1%, 95.6%, 94.1%, 95.6%,
94.5%, 94.5%, and 94.8%,
and 94.8%, and corresponding
corresponding average
average values valuesmodel
for SVM for SVM are
model
92.6%,are 92.6%,
92.0%, 92.0%,
92.3%, and92.3%,
92.7%.and 92.7%.(98.0%)
F1 score F1 score
and(98.0%) and(98.1%)
accuracy accuracyfor(98.1%) for the are
the RF model RF greater
model
are greater
than than corresponding
corresponding values
values for the KNN,for the KNN,
SVM, SVM,
and LDA and LDA
models. models.
Overall, the Overall,
results from the results
Table 5 from
show
Table 5 show
that the that the
RF model RF model
outclassed outclassed
the KNN, SVM, the and
KNN,LDASVM, and LDA
models models inthe
in predicting predicting
types of the mangotypes in
of mango
this study.in this study.

Appl. Sci. 2020, 10, x FOR PEER REVIEW Figure

Figure23.
23.Classification
Classificationof
ofRF
RFmodel.
model. 23 of 25

Figure
Figure 24.
24. The
Thecomparison
comparisonof
offour
fourmodels
models include
include LDA,
LDA, SVM,
SVM, KNN,
KNN, and
and RF.
RF.

Table 5. The accuracy of model.

Precision F1 Score
Type G1 G2 G3 Sensitivity (%) Accuracy (%)
(%) (%)
G1 348 2 0 99.4 98.6 99.0
Random Forest G2 5 305 7 96.2 97.8 97.0 98.1
G3 0 5 361 98.6 97.6 98.1
Linear G1 320 25 5 91.4 90.4 90.9 87.9
Appl. Sci. 2020, 10, 5775 23 of 25

Table 5. The accuracy of model.

Sensitivity Precision F1 Score Accuracy

Type G1 G2 G3
(%) (%) (%) (%)
G1 348 2 0 99.4 98.6 99.0
Random Forest G2 5 305 7 96.2 97.8 97.0 98.1
G3 0 5 361 98.6 97.6 98.1
Linear G1 320 25 5 91.4 90.4 90.9
Discriminant G2 30 253 34 79.8 83.8 81.8 87.9
Analysis (LDA) G3 4 24 338 92.3 89.7 91.0
G1 338 5 7 96.6 94.9 95.6
K-Nearest
G2 15 282 20 89.0 95.3 92.2 94.8
Neighbors (KNN)
G3 3 9 354 96.7 96.7 96.7
G1 336 7 7 96.0 93.1 94.6
Support Vector
G2 24 277 16 87.4 91.1 89.3 92.7
Machine (SVM)
G3 1 20 348 94.3 93.8 94.1

9. Conclusions
The classification was implemented with supporting the ML algorithms based on external features
and weights of mangoes and then using four ML models as LDA, SVM, KNN, and RF respectively to
grade mango automatically. There are several conclusions drawn as follows:

- In this study, the quality of mango is classified by four models (RF, LDA, KNN, and SVM).
The models have high accurate over 87.9%, especially the model of RF has the best predictive
precision of 98.1% compared with 3 remaining models. Therefore, classification based on rules
generated from input variables is suitable to classify the quality of mango.
- Throughout the classification, a sequence of analytical methods in computer vision is used to
transform the captured image of the mango to the image form that can be easily extracted feature
from the mango. The experiment shows that such methods are successful when the predicted
results have a small error.
- The solution of designed ML could maintain high prediction accuracy for different mangoes.
However, it should be applied similarly to the sample mangoes.
- This study classifies mangoes based on four features consist of length, width, defect, and weight.
The results showed that successful combination of extracted features as mangoes are classified
with 98.1% accuracy.
- This study indicated that data normalization and the elimination of outliers are essential to
enhance the grading accuracy of mango. The results demonstrated a significant increase in
the accuracy of the mango grading by over 37.5%. Furthermore, by optimizing its parameters,
the performance of models increased by more than 6.5% (except for the LDA model, which only
increased by 1.34%). Consequently, high accuracy is achieved not only by a good combination of
the extracted features but also by the properties of machine learning.

Author Contributions: Conceptualization, N.T.T.; methodology, N.T.T.; software, N.T.M.L.; validation, N.T.M.L.;
formal analysis, N.T.M.L.; resources, N.T.T.; data curation, N.T.M.L.; writing—original draft preparation, N.T.M.L.;
writing—review and editing, N.T.T.; visualization, N.T.M.L.; supervision, N.T.T.; project administration, N.T.T.;
funding acquisition, N.T.T. All authors have read and agreed to the published version of the manuscript.
Funding: This research was funded by Department of Science and Technology—Dong Thap province, Vietnam.
Acknowledgments: This research is supported by Ho Chi Minh City University of Technology and Education, Vietnam.
Conflicts of Interest: The authors declare no conflict of interest.
Appl. Sci. 2020, 10, 5775 24 of 25

References
1. Abbott, J.A. Quality measurement of fruits and vegetables. Postharvest Boil. Technol. 1999, 15, 207–225.
[CrossRef]
2. Nandi, C.S.; Tudu, B.; Koley, C. A Machine Vision Technique for Grading of Harvested Mangoes Based on
Maturity and Quality. IEEE Sens. J. 2016, 16, 6387–6396. [CrossRef]
3. Pise, D.; Upadhye, G.D. Grading of Harvested Mangoes Quality and Maturity Based on Machine Learning
Techniques. In Proceedings of the 2018 International Conference on Smart City and Emerging Technology
(ICSCET), Maharashtra, India, 5 January 2018; Institute of Electrical and Electronics Engineers (IEEE):
Piscataway, NJ, USA, 2018; pp. 1–6.
4. Pandey, R.; Gamit, N.; Naik, S. A novel non-destructive grading method for Mango (Mangifera Indica L.)
using fuzzy expert system. In Proceedings of the 2014 International Conference on Advances in Computing,
Communications and Informatics (ICACCI), Noida, India, 24–27 September 2014; Institute of Electrical and
Electronics Engineers (IEEE): Piscataway, NJ, USA, 2014; pp. 1087–1094.
5. Sa’Ad, F.; Ibrahim, M.F.; Shakaff, A.; Zakaria, A.; Abdullah, M. Shape and weight grading of mangoes using
visible imaging. Comput. Electron. Agric. 2015, 115, 51–56. [CrossRef]
6. Schulze, K.; Nagle, M.; Spreer, W.; Mahayothee, B.; Müller, J. Development and assessment of different
modeling approaches for size-mass estimation of mango fruits (Mangifera indica L., cv. ‘Nam Dokmai’).
Comput. Electron. Agric. 2015, 114, 269–276. [CrossRef]
7. Budiastra, I.W.; Punvadaria, H.K. Classification of Mango by Artificial Neural Network Based on Near
Infrared Diffuse Reflectance. IFAC Proc. Vol. 2000, 33, 157–161. [CrossRef]
8. Nandi, C.S.; Tudu, B.; Koley, C. Computer vision based mango fruit grading system. In Proceedings of the
International Conference on Innovative Engineering Technologies, Bangkok, Thailand, 28–29 December 2014.
9. Wanitchang, P.; Terdwongworakul, A.; Wanitchang, J.; Nakawajana, N. Non-destructive maturity classification of
mango based on physical, mechanical and optical properties. J. Food Eng. 2011, 105, 477–484. [CrossRef]
10. Zawbaa, H.M.; Hazman, M.; Abbass, M.; Hassanien, A.E. Automatic fruit classification using random forest
algorithm. In Proceedings of the 2014 14th International Conference on Hybrid Intelligent Systems, Hawally,
Kuwait, 14–16 December 2014; Institute of Electrical and Electronics Engineers (IEEE): Piscataway, NJ, USA,
2015; pp. 164–168.
11. Kumar, A.; Gill, G. Computer vision based model for fruit sorting using K-nearest neighbour classifier. Int. J.
Electr. Electron. Eng. 2015, 2, 1694–2426.
12. Nandi, C.S.; Tudu, B.; Koley, C. An automated machine vision based system for fruit sorting and grading.
In Proceedings of the 2012 Sixth International Conference on Sensing Technology (ICST), Kolkata, India,
18–21 December 2012; Institute of Electrical and Electronics Engineers (IEEE): Piscataway, NJ, USA, 2013;
pp. 195–200.
13. Bao, W.; Lai, W.-S.; Ma, C.; Zhang, X.; Gao, Z.; Yang, M.-H. Depth-Aware Video Frame Interpolation.
In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long
Beach, CA, USA, 15–20 June 2019; Institute of Electrical and Electronics Engineers (IEEE): Piscataway, NJ,
USA, 2019; pp. 3698–3707.
14. Jiang, H.; Sun, D.; Jampani, V.; Yang, M.-H.; Learned-Miller, E.; Kautz, J. Super SloMo: High Quality
Estimation of Multiple Intermediate Frames for Video Interpolation. In Proceedings of the 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; Institute
of Electrical and Electronics Engineers (IEEE): Piscataway, NJ, USA, 2018; pp. 9000–9008.
15. Ito, K.; Xiong, K. Gaussian filters for nonlinear filtering problems. IEEE Trans. Autom. Control. 2000, 45,
910–927. [CrossRef]
16. Faroudja, Y. NTSC and beyond [TV]. IEEE Trans. Consum. Electron. 1988, 34, 166–178. [CrossRef]
17. Sankur, B.; Sezgin, M. Survey over image thresholding techniques and quantitative performance evaluation.
J. Electron. Imaging 2004, 13, 146. [CrossRef]
18. Caselles, V.; Coll, T. A geometric model for active contours in image processing. Numer. Math. 1993, 66, 1–31.
[CrossRef]
19. Dhameliya, S.; Kakadiya, J.; Savant, R. Volume Estimation of Mango. Int. J. Comput. Appl. 2016, 143, 11–16.
[CrossRef]
Appl. Sci. 2020, 10, 5775 25 of 25

20. Charoenpong, T.; Kamhom, P.; Chaninongthai, K.; Krairiksh, M.; Chamnongthai, K. Volume measurement of
mango by using 2D ellipse model. In Proceedings of the 2004 IEEE International Conference on Industrial
Technology, IEEE ICIT ’04, Hammamet, Tunisia, 8–10 December 2004; Institute of Electrical and Electronics
Engineers (IEEE): Piscataway, NJ, USA, 2005; Volume 3.
21. Varma, S.; Simon, R. Bias in error estimation when using cross-validation for model selection. BMC Bioinform.
2006, 7, 91. [CrossRef] [PubMed]
22. Zill, D.G.; Wright, W.S.; Cullen, M.R. Advanced Engineering Mathematics; Jones & Bartlett Learning: Burlington,
MA, USA, 2011.
23. Breiman, L. Random forests. Mach. Learn. 2001, 45, 5–32. [CrossRef]
24. Shalev-Shwartz, S.; Ben-David, S. Decision Trees. In Understanding Machine Learning; Cambridge University
Press: Cambridge, UK, 2014.
25. Weinberger, K.Q.; Lawrence, K.S. Distance metric learning for large margin nearest neighbor classification.
J. Mach. Learn. Res. 2009, 10, 207–244.
26. Boltyanski, V.; Martini, H.; Soltan, V. The Kuhn–Tucker Theorem. In Geometric Methods and Optimization
Problems; Springer Science and Business Media: New York, NY, USA, 1999; pp. 78–92.
27. Hofmann, T.; Schölkopf, B.; Smola, A.J. Kernel methods in machine learning. Ann. Stat. 2008, 36, 1171–1220.
[CrossRef]
28. Forman, G. An extensive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res.
2003, 3, 1289–1305.
29. Willmott, C.; Matsuura, K. Advantages of the mean absolute error (MAE) over the root mean square error
(RMSE) in assessing average model performance. Clim. Res. 2005, 30, 79–82. [CrossRef]
30. De Myttenaere, A.; Golden, B.; Le Grand, B.; Rossi, F. Mean Absolute Percentage Error for regression models.
Neurocomputing 2016, 192, 38–48. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).