Postharvest Biology and Technology 166 (2020) 111201

Contents lists available at ScienceDirect

Postharvest Biology and Technology

journal homepage: www.elsevier.com/locate/postharvbio

Evaluation of image processing technique as an expert system in mulberry T

fruit grading based on ripeness level using artificial neural networks (ANNs)
and support vector machine (SVM)
Hossein Azarmdela, Ahmad Jahanbakhshib, Seyed Saeid Mohtasebia,*, Alfredo Rosado Muñozc
a
Department of Agricultural Machinery Engineering, Faculty of Agricultural Engineering and Technology, University of Tehran, Karaj, Iran
b
Department of Biosystems Engineering, Faculty of Agriculture and Natural Resources, University of Mohaghegh Ardabili, Ardabil, Iran
c
Department of Electronic Engineering, University of Valencia, Burjassot, Valencia, Spain

A R T I C LE I N FO A B S T R A C T

Keywords: Image processing and artificial intelligence (AI) techniques have been applied to analyze, evaluate and classify
Mulberry mulberry fruit according to their ripeness (unripe, ripe, and overripe). A total of 577 mulberries were graded by
Maturity detection an expert and the images were captured by an imaging system. Then, the geometrical properties, color, and
Classification texture characteristics of each segmented mulberry was extracted using two feature reduction methods:
Machine vision
Correlation-based Feature Selection subset (CFS) and Consistency subset (CONS). Artificial Neural Networks
Artificial intelligence
(ANN) and Support Vector Machine (SVM) were applied to classify mulberry fruit. ANN classification with the
CFS subset feature extraction method resulted in the accuracy of 100 %, 100 %, and 99.1 % and the least mean
square error (MSE) values of 9.2 × 10-10, 3.0 × 10-6, and 2.9 × 10-3 for training, validation, and test sets, re-
spectively. The ANN structure with the CONS subset feature extraction method resulted in the acceptable model
with the accuracy of 100 %, 98.9 %, and 98.3 % and calculated MSE values of 4.9 × 10-9, 3.0 × 10-3, and
3.1 × 10-3 for training, validation, and test sets, respectively. In general, the machine vision system combined
with the ANN and SVM algorithms successfully classified mulberries based on maturity. Finally, the ANN model
with four features (R, B, b*, and Cr) selected through the CONS subset method with the least number of inputs
and acceptable high classification accuracy with low MSE value was proposed as the proper model for online
applications.

1. Introduction Liming and Yanchao (2010) found that size detection error using
image processing for strawberries was less than 5 %, color classification
Mulberry fruit are usually harvested manually by shaking the tree accuracy was 89 %, and shape classification accuracy was higher than
branches or mechanically using a shaker, resulting in a mixture of un- 90 %. Mohammadi et al. (2015) found differences among different le-
ripe, ripe, and overripe fruit. Different stages of ripeness result in low vels of ripeness of persimmons for most of the visual characteristics,
marketability and increased product waste. An essential post-harvest including the R, G, and B channels, and the greyscale, and that the
operation is the agricultural product classification based on their ex- Quadratic Differential Analysis (QDA) was able to grade the fruit with
ternal features. 90 % overall accuracy. Wan et al. (2018) used computer vision and the
Machine vision, combined with artificial intelligence, has attracted backpropagation neural network method to detect three ripeness levels
a great deal of attention for use in agricultural and food industry ap- of tomato. Khojastehnezhad et al. (2019) used image processing to
plications, especially for recognition and classification of quality factors detect differences of apricots with respect to different features such as
(Bargoti and Underwood, 2017; Teimouri et al., 2018; Azarmdel et al., G, grayscale, L* and b*. In addition, the LDA and QDA techniques were
2019; Khojastehnazhand et al., 2019; Taheri-Garavand et al., 2019). able to classify apricots with 90.4 % and 92.3 % accuracies, respec-
Image processing and machine vision provides us with new knowledge tively.
that can be applied to control the quality of food and agricultural In addition to classification, the SVM algorithm can be used for both
products with high accuracy (Nogales-Bueno et al.; 2014; Mohammadi estimation and classification. Applications of the SVM algorithm can
et al.; 2015; Fashi et al., 2019). include pattern recognition, image processing and medical use, (Nouri

⁎
Corresponding author.
E-mail address: [email protected] (S.S. Mohtasebi).

https://doi.org/10.1016/j.postharvbio.2020.111201
Received 25 December 2019; Received in revised form 29 March 2020; Accepted 29 March 2020
Available online 18 April 2020
0925-5214/ © 2020 Elsevier B.V. All rights reserved.
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

et al., 2019; Ebrahimi et al., 2017; Peng et al., 2018). SVM has been Fig. 3.
applied for pattern and texture recognition, harvest time estimation, As shown in Fig. 3, to segment each single mulberry class, any of the
and product quality determination. Yu et al. (2014) proposed a new R, G, and B channels can be used, but in the combined samples with
method for detecting fresh jujube cracks and maturity stages using different levels of ripeness, an acceptable segmentation cannot be
ultra-high-resolution imaging in combination with image processing. achieved in R and G color channels. Therefore, among the studied color
They used differentiation models to identify mature and cracked fruit. channels, channel B showed the best segmentation for all samples. In
Mizushima and Lu (2013) developed an automatically adjustable al- the next step, any possible holes in the images were filled. In addition,
gorithm that graded the apple fruit based on its apparent color using the some small objects rather than the fruit were observed beside the
SVM and the Otsu thresholding method. In another study, Sun et al. samples (non-target objects), which were also removed to avoid any
(2014) determined the position and type of whiteness in rice grains disturbance in the feature extraction process.
with high accuracy by combining the SVM and image processing
methods. A great deal of research has been conducted on the grading of 2.2. Separating the stem ends from the mulberries
agricultural products such as mango (Vélez-Rivera et al., 2014), apple
(Jarolmasjed et al., 2016), potato (Al Riza et al., 2017), olive (Beyaz To accurately classify the mulberries and reduce the errors in the
et al., 2017), banana (Larada et al., 2018), date (Nasiri et al., 2019), classification process, it is necessary to segment the mulberry area
carrot (Jahanbakhshi and Kheiralipour, 2019), and pomegranate (Fashi without any stems. Thus, after the initial segmentation, an erosive op-
et al., 2019) using machine vision. erator was used to separate the stems from the fruit (Fig. 4). This is
The present study investigates the capability of the image proces- performed by considering a structural element indicating the number of
sing method and classification techniques in grading mulberries based pixels in the neighborhood to filter the image by eroding the edges of
on their apparent characteristics. In this research, the emphasis is on the objects (Solomon and Breckon, 2011). Afterward, the image was
the applicability of machine vision as a quick, inexpensive and accurate dilated to return the mulberry area to the initial size with the least
tool for grading mulberries in accordance with their level of ripeness. possible erosion.
Thus, image processing algorithms and classification methods (ANN
and SVM) were applied to extract mulberry features to classify the fruit 2.3. Extraction of geometrical, color and texture features of mulberries
in six classes with three ripeness levels (unripe, ripe, and overripe).
Finally, a proposed grading system is presented, indicating the feasi- After the initial segmentation process, each fruit in the image was
bility of applying machine vision and artificial intelligence for mulberry isolated from the initial image to extract the related features. Fig. 5
fruit classification. illustrates the smallest rectangular banding boxes around the samples in
which the inside area is used to separate the color samples for the
2. Materials and methods subsequent process.
As it is necessary to extract mulberry features apart from the rest of
In this study, white (Morus alba) and red (Morus rubra) mulberry the image, the color image is overlaid on the binary image. Therefore,
types were freshly harvested from an orchard in Tabriz - Iran. First, an the coordinates of the mulberry samples in the primary color image,
expert graded samples of each type into three groups of unripe, ripe, which match the position of "1s" (white area) of the binary image, are
and overripe fruit and the images of all six classes were captured. In conserved, and the rest is converted to “0” (black area). Finally, the
total, 106 unripe, 92 ripe, and 100 overripe fruit were in the set of geometrical, texture, and color characteristics are determined for each
white mulberry, and 80 unripe, 109 ripe, and 90 overripe fruit were in mulberry sample. Features extracted from the mulberries are reported
the group of red mulberry (Figs. 1 and 2). In this study, the image ac- in Table 1.
quisition system had a lighting case that consisted of 432 white and
yellow LED lights installed inside the case. A Casio EX-H20 G 14.1 M P 2.4. Feature selection
CCD digital camera was used to capture the images.
Modern devices for grading the agricultural products capture groups CFS subset (CFS) and Consistency subset (CONS) methods were used
of samples in a single frame. By capturing each sample in one snapshot, to extract the most relevant features from the images. Using the CFS
the time spent for capturing and analyzing the mulberry samples will method, 14 top features were selected based on the ‘Best First’ criterion.
increase, and the capacity of the sorting system will decrease. Therefore Four top features were also selected using the CONS method. Initial and
to introduce the image processing algorithm for an online grading selected features based on both methods are presented in Table 1.
system, it is necessary to capture the samples all at once. Therefore, one WEKA software was used to reduce the input data dimension.
image was obtained from all the samples of each mulberry class.
2.5. Classification algorithms
2.1. Selection of the appropriate color channel and mulberry segmentation
2.5.1. Artificial Neural Network (ANN)
One of the most common methods for image segmentation is to Neural networks are simple modeling of real nervous systems. The
apply the Otsu (1979) thresholding method. This algorithm converts scope of application of these networks is so broad that it includes ap-
the initial image into two classes (foreground and background) by plications from classification to those such as estimation and detection.
minimizing the intra-class variances. By comparing the pixel intensities An ANN is a set of simple structures (neurons, also called nodes) that
and the corresponding number of the pixels, the minimum value of the together describe a final complex system. A set of nodes and connec-
histogram is considered as the segmentation criteria. Note that, as tions between them are the two main components of a network. Each
shown in Fig. 3, if each mulberry type (white or red) was considered node is a network computing unit that can perform simple or complex
separately, any of the R, G, or B color channels could be used for image processing on the inputs. Unlike mathematical models, neural network
segmentation. However, the classification operation for mulberries with models can detect the relationships among parameters without ex-
different levels of ripeness (unripe, ripe, and overripe) was performed tracting those relationships, and this has made the neural networks a
for both types. Besides, to ensure the capability of the Otsu method to powerful tool in classification and modeling (Jahanbakhshi and Salehi,
segment the fruit, the proposed algorithm was applied to classify a 2019; Utai et al., 2019). To run the network, 65 % of all the input data
combination of samples with different levels of ripeness. Images of R, G, were considered as training data, 15 % as validation, and 20 % as test
and B color channels for separated (left and right with smaller size) and data. We applied MATLAB 2017 software to process the images and run
combined white and red fruit (center with bigger size) are shown in the artificial intelligent models.

2
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Fig. 1. Six different categories of two mulberry types (white and red).

In the next step, conventional hyperbolic tangent sigmoid (tansig) 2

f (x ) = −1
and logarithm sigmoid (logsig) transfer functions were used (Marti 1 + e−2x (1)
et al., 2013) to activate the weighted outputs. The following are the
1
equations related to the tangent sigmoid and the logarithm sigmoid f (x ) =
1 + e−x (2)
activation functions.
In the hidden layer, a set of 24 neurons ranging from 2 to 25 was
tried. To show the statistical reliability and develop an accurate model

Fig. 2. Imaging system and mulberry classification.

3
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Fig. 3. Segmenting mulberry samples using the Otsu method in R, G and B color channels for red mulberries (left), white mulberries (right) and combined types
(center).

on the selected structure of the ANN, researchers have tested different Based on the two training algorithms (‘trainlm’ and ‘traingdx’), two
numbers of replications. Each structure was trained 50 times and the input data sets (CFS and CONS), and four different conditions resulted
best value was recorded (Teimouri et al., 2016). The ANN structure and from the combination of two activation functions (Tansig and Logsig) in
applied models are presented in Fig. 6. hidden and output layers, a total of 16 running structures were created.

4
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Fig. 4. Omitting the mulberry stem ends using edge erosion and dilation algorithm: A) mulberry samples with stem ends, and B) final binary image without stem
ends.

By taking into account the number of 24 neurons in the hidden layer generalized to nonlinear states. By using this algorithm, data are cate-
and repeating the same structure for 50 times, a total of 1,200 gorized into two phases of train and test sets, and to validate the data,
(24 × 50) iterations were conducted for each of the 16 structures. A cross-validation techniques such as k-fold, holdout, or leave-n-out
total of 19,200 (1200 × 16) runs were conducted to judge and select training methods are applied. In order to find the optimal value of the
the best ANN structure suited for mulberry classification based on the penalty parameter (C) and the kernel parameter (σ), the holdout cross-
least mean MSE value. The best ANN selection process is presented in validation was applied to perform the classification with the highest
Fig. 7. accuracy. For this purpose, 20 % of the whole data was considered as
test data each time, and the rest was taken as training data.
SVM is a binary classification model. This algorithm converts the
2.5.2. Support Vector Machine (SVM) training data space into a higher dimension through nonlinear mapping
The SVM algorithm is one of the most accurate and robust algo- and then looks for a hyperplane in this new dimension to separate
rithms in data mining and can be used to classify linear and nonlinear samples of one class from the other classes. This algorithm uses con-
data. The SVM algorithm was first proposed by Cortes and Vapnik cepts such as support vectors and margins to find this hyperplane. A
(1995) to classify data that were linearly separable and later

Fig. 5. Separating the mulberries in the initial image as the individual samples by matching the white pixels “1s” of the binary image with the initial color image
based on the rectangular bounding box.

5
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Table 1
Extracted and selected features using CFS subset (CFS) and Consistency subset (CONS) methods.
Total extracted features Selected features based on evaluation method

CFS subset (CFS)

Color features (Mean) Geometric Features Texture features Color features (Mean) Geometric Features Texture features
R-(RGB) Area Standard deviation G-(RGB) Perimeter Contrast
G-(RGB) Perimeter Entropy B-(RGB) Equivalent Diameter Homogeneity
B-(RGB) Eccentricity Contrast Gray Area/Perimeter
Gray Extent Homogeneity L-(La*b*)
L*-(La*b*) Orientation Correlation a*-(La*b*)
a*-(La*b*) Equivalent Diameter Energy b*-(La*b*)
b*-(La*b*) Max axis/Min axis M- (CMY)
H-(HSV) Area/Perimeter Cb-(YCbCr)
S-(HSV) Roundness Cr-(YCbCr)
V-(HSV)
C- (CMY)
M- (CMY) Consistency subset (CONS)
Y-(CMY)
Y-(YIQ) R-(RGB)
I-(YIQ) B-(RGB) —————————— ——————————
Q-(YIQ) b*-(L*a*b*)
Cb-(YCbCr) Cr-(YCbCr)
Cr-(YCbCr)

Fig. 6. The ANN structure with the number of the input features, the number of the nodes in the hidden layer, the activation functions and the training algorithms.

sequence of binary clusters was used to categorize several classes. functions, polynomial, or linear functions. Therefore, the MSE values
Assuming that each of the six classes used in this study are linearly were calculated along with the classification accuracy to find the op-
separable in pairs, the SVM binary class was trained for 15 times as a timal 'C' and 'σ' values. Wu et al (2018) used RBF kernels in their study
permutation of T62 by the SVM algorithm, and the final result was de- with (C, σ) = {10, 20, …, 100} × {1, 1.5, 2} and polynomials with (C,
termined based on the sum of the results in each voting (Wu et al., p) = {1, 2, …, 100} × {2, 3}. Since in SVM it is important to find the
2018; Mohammadi and Minaei, 2019). appropriate values of 'C' and 'σ' to define the best network structure
To classify nonlinearly separable data, the data are mapped onto with the highest classification rate and the lowest error rate, the in-
space with larger dimensions (feature space). Such algorithms require tervals with smaller steps with (C, σ) = {1, 2, …, 100} × {0.25, 0.5, …,
complex computation, which results in slow functioning. Therefore, 2.5} and polynomials with (C, p) = {1, 2, …, 100} × {2, 3} were tried
kernel functions are used to separate nonlinear data linearly so that in this study. The linear kernel was also applied to evaluate the clas-
based on these transformations, an optimal boundary could be drawn sification accuracy and compare it with the results of RBF and poly-
for separation, which is not necessarily a straight line. The advantage of nomial kernels. In general, an algorithm with 10400 iterations was
these kernels is creating connections between data points and simpli- implemented to extract the optimal 'C' and 'σ' values using input fea-
fying the complex computations. tures of both CFS and CONS methods.
Some of the kernels used in SVM are radian basis or Gaussian

6
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Fig. 7. The process of selecting the best ANN structure for mulberry classification.

2.6. Comparing the results respectively (Marti et al., 2013). To train the network using the SVM
algorithm, the input data was normalized between [0 – 1].
After the network structure was determined, the data were imported
to the neural networks. To normalize the input data in the [-1 − 1] and Yi − Ymin
YN =
[0 – 1] intervals, depending on the activation function (tansig and Ymax − Ymin (3)
logsig) of the network output, equations (3) and (4) were used

7
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Yi − Ymin studied, the best structure with the least error value was selected as the
YN = 2 × ( )−1
Ymax − Ymin (4) optimal structure. Fig. 9 shows the minimum error values among all 16
models. The values are presented for 16 different models, resulting from
In these relations, Yi represented the input data, Ymin was the
the combination of activation functions (TAN and LOG), network
minimum input data, Ymax was the maximum input data and YN showed
learning algorithms (LM and GDX), and input selection methods (CONS
the normalized data in the given interval.
and CFS).
To compare the performance of the ANN and SVM algorithms, the
In all network structures, the error value calculated by using the CFS
Mean Square Error (MSE) was calculated as described in equation (5)
feature extraction method is lower than the value obtained through the
(Emamgholizadeh et al., 2015).
CONS method (Fig. 9), which may be due to a large number of the
n
∑i = 1 (oi − pi )2 features in the CFS method. The higher the number of network inputs,
MSE =
n (5) the higher the computational burden on the network. If the final clas-
sification function (optimal structure) is used in any online processing
In this relation, n, oi, and pi were the number of data, the values
systems (machines used to process mulberries), the large number of
observed, and the values predicted, respectively.
inputs will increase the computation time. The best network structure
In order to evaluate the accuracy of the given algorithm, three
based on the lowest MSE using CFS and CONS methods was the LM-T-L
statistical parameters, namely sensitivity (SEN), specificity (SPE) and
and LM-L-L structures with values of 1.3 × 10-3 and 2.1 × 10-3, re-
accuracy (ACC), were used. Each parameter was calculated as described
spectively. It should be mentioned that the difference in error values
in equations (6), (7), and (8), respectively (Omid, 2011).
between these two cases is close to each other, which shows the se-
TP lected four features with the CONS method can be considered as an
Sensitivity =
(TP + FN ) (6) acceptable model compared to 14 features selected with the CFS
method. This is discussed in the following, and the results are presented
TN
Specificity = in detail.
(TN + FP ) (7) Table 2 presents the confusion matrix for the best ANN structure
(TP + TN ) using the CFS method. The results of the training and validation data
Accuracy = show that all samples are correctly classified in their groups with 100 %
(TP + TN + FN + FP ) (8)
accuracy. The sensitivity of the BR1 resulted in 94.44 % for the test set.
In these relations, True Positives (TP) are the number of samples On the other hand, the specificity of BR2 resulted in 98.85 %, which
that are correctly classified in group A. True Negatives (TN) are the shows that misclassified samples of BR3 fall in this group. The total
number of samples belonging to other groups. False Negatives (FN) are accuracy of the network for the test set resulted in 99.13 %. The MSE
the number of samples belonging to group A which are incorrectly values of 9.2 × 10-10, 3.0 × 10-6, and 2.9 × 10-3 are calculated for the
classified in other groups. False Positives (FP) are the number of sam- training, validation, and test sets, respectively.
ples of other groups that are incorrectly classified as group A. The neural network results using the CONS subset feature extraction
method are reported in Table 3. Using this subset for classification
3. Results and discussion showed the sensitivity, specificity, and accuracy of 100 % for the
training dataset. The sensitivity value of 94.44% was calculated for
3.1. ANN results BW2 in the validation dataset. On the other hand, the specificity value
of the BR1 resulted in 98.59 %, indicating that the misclassified samples
Output data for fruit were divided into three categories (training, were fallen in this group.
validation, and test data). To evaluate the performance of activation The results of the test set showed the sensitivity values of 95.24%
functions in fruit classification, two activation functions were tried in and 92.31% for BW3 and BR1, respectively. On the other hand the
the hidden and output layers to train the neural network: tangent sig- specificity value of 98.96 % and 98.94 % were calculated for the BW2
moid (TAN) and logarithm sigmoid (LOG). and BW3, respectively. Besides, the total accuracy of the test set re-
To evaluate the network structure and to ensure its reliability, 50 sulted in 98.26 %. The MSE values of 4.9 × 10-9, 3.0 × 10-3, and
replications were assigned to each node in the network structure. The 3.1 × 10-3 were obtained for the training, validation, and test sets, re-
mean MSE values of training, validation, and test sets were determined spectively.
for each of the replicates. Then, the best network structure was selected Results show that the average of MSE for 50 replications using the
based on the lowest total mean MSE value among the tried number of CFS feature selection method was lower than that of the CONS method
the nodes (2-25 nodes). By considering the combination of the LOG and (Fig. 9). Still, the results indicate a more accurate classification of dif-
TAN activation functions in both hidden and output layers, the results ferent groups of mulberries through the CONS method (Table 3) with
for the four modes were obtained. As mentioned above, two sets of lower error values. Although the average MSE value of CFS based se-
different inputs were introduced to the network based on the CFS and lected features is lower than that of CONS, the related structure trained
CONS feature selection methods. Four graphs containing four lines each with the CONS subset features showed lower MSE value as we have
of which represents one of the network structures are shown in Fig. 8. reported the lowest MSE error of each structure among 50 tried re-
Figs. 8A and 8C are related to the network training based on the plications. Although, the selected representative structure of the LM-T-L
Levenberg-Marquardt (LM) algorithm. Figs. 8B and 8D belong to the CFS subset resulted in lower MSE value compared to the LM-L-L of
gradient descend training (GDX) algorithm. The analyses show higher CONS subset, considering a network structure with lower input features
MSE values in the LOG-TAN (logarithm sigmoid activation function in has priority over a large number of the features.
the hidden layer and tangent sigmoid activation function in the output
layer) network structure. Therefore, using the TAN function in the 3.2. SVM results
output layer results in higher error values compared to the other modes.
The results also show that using the LM algorithm (Figs. 8A and 8C), the Table 4 presents the results of the classification of six groups of fruit
MSE values face many fluctuations (red line) without a constant trend by SVM algorithm using two different feature selection methods (CFS
by increasing the number of neurons in the hidden layer. The lowest subset and Consistency subset) as well as the MSE values and the
error values in all four figures belong to the TAN-LOG and LOG-LOG classification values of training and test sets using RBF, polynomial, and
structures (blue and brown lines) (Fig. 8). linear kernel functions. The optimal values of 'C' and 'σ' for using dif-
After selecting the appropriate network structure from all the cases ferent modes are also presented in Table 4. The least error values by

8
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Fig. 8. Results of mean square error (MSE) for different ANN topologies using the Levenberg Marquardt (LM) and gradient descend training (GDX) algorithms in both
(CFS) and (CONS) subsets. Four different network structures based on the activation function of the nodes (logarithm sigmoid (LOG) or tangent sigmoid (TAN)) in
both hidden and output layers were tried.

using the CFS subset and CONS subset belong to the RBF and poly- CONS subset using the polynomial kernel are 1.7 × 10-2 and 1.8 × 10-2
nomial kernels, respectively. It should be noted that the polynomial for the training and test sets, respectively. Among both of the subsets,
functions with p = 2 are the same functions known as quadratic func- using the RBF kernel function with the CFS subset feature extraction
tions. method showed the lowest MSE. The classification accuracies con-
The SVM confusion matrix for the CFS and CONS subsets is given in cerning the CFS subset mode were 99.78 % and 99.12 % for training
Table 5. In addition to the low error rate of the CFS subset method with and test sets, respectively. These values for polynomial function using
an RBF kernel function, the sorting accuracy is also higher compared to the CONS subset were 99.57 % and 98.25 %.
the CONS subset with a polynomial function. In fact, among all the The results of the current study show that the automated vision
combinations under study, the best SVM network structure for sorting system can successfully classify fruit into three maturity levels based on
fruit is the CFS subset with an RBF kernel (Table 4). The MSE values of the color, shape, and texture using ANN and SVM classifiers. Among
the CFS and CONS subsets show the lowest MSE values of 9 × 10-3 and them, the best classification resulted in the ANN classifier using CONS
9 × 10-3 for training and test sets, respectively. The MSE values of the subset, with the least number of inputs, low error value, and acceptable

9
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

4. A proposed plan for mulberry classification

Fig. 10 shows the proposed plan of a sorting system for mulberry

classification. Fruit move into parallel grooves by the initial conveyor.
Also, the inclined position of the vibrating plate guides the fruit
downwards. The movement of the second conveyor belt carries the fruit
over the holes in the belt as they pass on in front of a camera installed
along with the device. After detecting the mulberry type through the
machine vision and processing section, the machine orders the specific
mulberry shooter by compressed air (pneumatic) to shoot the mulberry
towards the proper quality spot at the right time. Among the six mul-
berry classes, the sixth one falls into its box from the end of the con-
veyor belt without shooting. This group can be used for one of the
sound classes like BW2 or BR2. The other sound class can be guided
from the shortest channel with the minimum amount of air pressure.
The holes (seats) in the second conveyor belt are made so that samples
are placed in stable positions. Moreover, there are holes designed to let
the airflow contact over mulberry samples. The airflow rate is adjusted
to avoid any bruise or crush on mulberry fruit.
Fig. 9. Total Mean Square Error (MSE) values of 24 network structures with 50
replications per node. Each term in the horizontal axis represents the applied 5. Conclusion
functions: applied training algorithm (LM or GDX) - activation function of the
hidden layer (TAN or LOG) - activation function of the output layer (TAN or In this study, white and red mulberry fruit were classified according
LOG).
to maturity stage using image processing and artificial intelligence
classification algorithms. First, mulberry image segmentation was per-
Table 2 formed using the RGB color space. Among the tested color channels, the
Confusion matrix of the Artificial Neural Networks (ANN) using the CFS subset channel 'B' was selected as the best channel to classify fruit into three
(CFS) feature selection method with 20 neurons in the hidden layer (14-20-6). unripe, ripe, and overripe categories. In the next step, color, geometric,
Train Validation Test and texture features were extracted with two feature selection methods,
namely CFS and CONS. After the image processing step, feature ex-
SEN SPE ACC SEN SPE ACC SEN SPE ACC traction, and dimension reduction, ANN and SVM were applied to
(%) (%) (%) (%) (%) (%) (%) (%) (%)
classify each fruit as one of the six possible classes. Comparing the
BW1 100 100 100 100 100 100 100 100 99.13 performance of the two methods (ANN and SVM), the ANN showed a
BW2 100 100 100 100 100 100 significant advantage over the SVM for the mulberry classification. The
BW3 100 100 100 100 100 100 best classification performance was obtained by using the CFS subset
BR1 100 100 100 100 94.44 100
feature extraction method (14 selected features) with ANN. Although
BR2 100 100 100 100 100 98.85
BR3 100 100 100 100 100 100 the CFS method showed the least error value with the highest classifi-
MSE 9.2 × 10-10 3.0 × 10-6 2.9 × 10-3 cation rate, the final model was proposed based on both error value and
number of inputs. Since this research aims to develop a combined
Note: Mean Square Error (MSE); Sensitivity (SEN); Specificity (SPE); Accuracy machine vision and artificial intelligence algorithm for an online ap-
(ACC); three classes of white mulberry (BW1-3); three classes of red mulberry plication, therefore, it is proposed to use the CONS feature subset to
(BR1-3).
reduce the dimension of the initial data and introduce them to the ANN
with the LM-L-L (Levenberg-Marquardt-Logsig-Logsig) structure to ob-
Table 3 tain acceptable classification with low MSE value and the least number
Confusion matrix of Artificial Neural Networks (ANN) using the Consistency
of inputs. Thus, this structure is proposed to be used in a six-class
subset (CONS) feature selection method with 15 neurons in the hidden layer (4-
mulberry sorting system: two types and three ripeness levels. According
15-6).
to the results of this study, machine vision can be used as an accurate,
Train Validation Test fast, and non-destructive method to classify and grade mulberry fruit,
reduce product loss and increase its marketability.
SEN SPE ACC SEN SPE ACC SEN SPE ACC
(%) (%) (%) (%) (%) (%) (%) (%) (%)
Declaration of Competing Interest
BW1 100 100 100 100 98.59 98.85 100 100 98.26
BW2 100 100 94.44 100 100 98.96
The authors have declared no conflict of interest.
BW3 100 100 100 100 95.24 98.94
BR1 100 100 100 100 92.31 100
BR2 100 100 100 100 100 100 CRediT authorship contribution statement
BR3 100 100 100 100 100 100
MSE 4.9 × 10-9 3.0 × 10-3 3.1 × 10-3 Hossein Azarmdel: Methodology, Investigation, Software, Writing
- original draft, Formal analysis. Ahmad Jahanbakhshi:
Note: Mean Square Error (MSE); Sensitivity (SEN); Specificity (SPE); Accuracy
(ACC); three classes of white mulberry (BW1-3); three classes of red mulberry Conceptualization, Methodology, Investigation, Writing - original draft,
(BR1-3). Data curation. Seyed Saeid Mohtasebi: Supervision, Data curation,
Validation, Writing - review & editing. Alfredo Rosado Muñoz: Data
high classification accuracy. Therefore, the proposed algorithm could curation, Validation, Writing - review & editing.
objectively, rapidly, and effectively classify mulberry fruit by using a
non-destructive vision technique. Appendix A. Supplementary data

Supplementary material related to this article can be found, in the

10
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

Table 4
Results of Support Vector Machine (SVM) for mulberry classification with two sets of selected features (CFS subset (CFS) and Consistency subset (CONS)) using three
different kernel functions (RBF, polynomial, and linear).
Applied kernels Number of iterations MSE Accuracy (%) Poly order Optimized parameters
Train Test Train Test C σ
CFS subset (CFS)
RBF 100 × 4 × 10 0.009 0.009 99.78 99.12 ——— 1 1.75
polynomial 100 × 4 ×2 0.000 0.114 100.00 98.25 2 1 ———
Linear 100 × 4 0.009 0.035 99.78 99.12 1 5 ———
Consistency subset (CONS)
RBF 100 × 4 × 10 0.071 0.149 99.35 98.25 ——— 1 1
polynomial 100 × 4 ×2 0.017 0.018 99.57 98.25 2 2 ———
Linear 100 × 4 0.073 0.079 99.57 99.12 1 9 ———

Note: Mean Square Error (MSE); Radial Basis Function (RBF); penalty parameter (C); kernel parameter (σ).

Table 5
Confusion matrix of Support Vector Machine (SVM) using CFS subset (CFS) and consistency subset (CONS).
CFS subset (CFS) Consistency subset (CONS)

Train Test Train Test

Class SEN (%) SPE (%) ACC (%) SEN (%) SPE (%) ACC (%) SEN (%) SPE (%) ACC (%) SEN (%) SPE (%) ACC (%)
BW1 100 100 99.78 100 100 99.12 100 100 99.57 100 100 98.25
BW2 100 100 94.74 100 100 100 90.00 100
BW3 100 100 100 98.95 100 100 100 97.92
BR1 100 99.73 100 100 98.86 99.73 100 100
BR2 100 100 100 100 100 100 100 100
BR3 98.46 100 100 100 98.44 99.75 100 100
MSE 9.0 × 10-3 9.0 × 10-3 1.7 × 10-2 1.8 × 10-2

Note: Mean Square Error (MSE); Sensitivity (SEN); Specificity (SPE); Accuracy (ACC); three classes of white mulberry (BW1-3); three classes of red mulberry (BR1-3).

Fig. 10. The design of the proposed system for mulberry fruit classification.

online version, at doi:https://doi.org/10.1016/j.postharvbio.2020. References

111201.
Al Riza, D.F., Suzuki, T., Ogawa, Y., Kondo, N., 2017. Diffuse reflectance characteristic of
potato surface for external defects discrimination. Postharvest Biology and
Technology 133, 12–19. https://doi.org/10.1016/j.postharvbio.2017.07.006.
Azarmdel, H., Mohtasebi, S.S., Jafari, A., Muñoz, A.R., 2019. Developing an orientation

11
H. Azarmdel, et al. Postharvest Biology and Technology 166 (2020) 111201

and cutting point determination algorithm for a trout fish processing system using 123–128. https://doi.org/10.1016/j.scienta.2014.12.037.
machine vision. Computers and Electronics in Agriculture 162, 613–629. https://doi. Nasiri, A., Taheri-Garavand, A., Zhang, Y.D., 2019. Image-based deep learning automated
org/10.1016/j.compag.2019.05.005. sorting of date fruit. Postharvest Biology and Technology 153, 133–141. https://doi.
Bargoti, S., Underwood, J.P., 2017. Image segmentation for fruit detection and yield org/10.1016/j.postharvbio.2019.04.003.
estimation in apple orchards. Journal of Field Robotics 34 (6), 1039–1060. https:// Nogales-Bueno, J., Hernández-Hierro, J.M., Rodríguez-Pulido, F.J., Heredia, F.J., 2014.
doi.org/10.1002/rob.21699. Determination of technological maturity of grapes and total phenolic compounds of
Beyaz, A., Özkaya, M.T., İçen, D., 2017. Identification of some spanish olive cultivars grape skins in red and white cultivars during ripening by near infrared hyperspectral
using image processing techniques. Scientia Horticulturae 225, 286–292. https://doi. image: A preliminary approach. Food Chemistry 152, 586–591. https://doi.org/10.
org/10.1016/j.scienta.2017.06.041. 1016/j.foodchem.2013.12.030.
Cortes, C., Vapnik, V., 1995. Support-vector networks. Machine learning 20 (3), 273–297. Nouri, B., Mohtasebi, S.S., Jahanbakhshi, A., 2019. Application of an olfactory system to
https://doi.org/10.1007/BF00994018. detect and distinguish bitter chocolates with different percentages of cocoa. Journal
Ebrahimi, M.A., Khoshtaghaza, M.H., Minaei, S., Jamshidi, B., 2017. Vision-based pest of Food Process Engineering 42 (7), e13248. https://doi.org/10.1111/jfpe.13248.
detection based on SVM classification method. Computers and Electronics in Omid, M., 2011. Design of an expert system for sorting pistachio nuts through decision
Agriculture 137, 52–58. https://doi.org/10.1016/j.compag.2017.03.016. tree and fuzzy logic classifier. Expert Systems with Applications 38 (4), 4339–4347.
Emamgholizadeh, S., Parsaeian, M., Baradaran, M., 2015. Seed yield prediction of sesame https://doi.org/10.1016/j.eswa.2010.09.103.
using artificial neural network. European Journal of Agronomy 68, 89–96. https:// Otsu, N., 1979. A threshold selection method from gray-level histograms. IEEE
doi.org/10.1016/j.eja.2015.04.010. Transactions on Systems, Man, and Cybernetics 9 (1), 62–66.
Fashi, M., Naderloo, L., Javadikia, H., 2019. The relationship between the appearance of Peng, H., Shao, Y., Chen, K., Deng, Y., Xue, C., 2018. Research on Multi-class Fruit
pomegranate fruit and color and size of arils based on image processing. Postharvest Recognition Based on Machine Vision and SVM. IFAC-PapersOnLine 51 (17),
Biology and Technology 154, 52–57. https://doi.org/10.1016/j.postharvbio.2019. 817–821. https://doi.org/10.1016/j.ifacol.2018.08.094.
04.017. Solomon, C., Breckon, T., 2011. Fundamentals of Digital Image Processing: A practical
Jahanbakhshi, A., Kheiralipour, K., 2019. Carrot sorting based on shape using image approach with examples in Matlab. John Wiley & Sons.
processing, artificial neural network, and support vector machine. Journal of Sun, C., Liu, T., Ji, C., Jiang, M., Tian, T., Guo, D., Liang, 2014. Evaluation and analysis
Agricultural Machinery 9 (2), 295–307. https://doi.org/10.22067/jam.v9i2.70579. the chalkiness of connected rice kernels based on image processing technology and
(In Persian). support vector machine. Journal of Cereal Science . 60 (2), 426–432. https://doi.org/
Jahanbakhshi, A., Salehi, R., 2019. Processing watermelon waste using Saccharomyces 10.1016/j.jcs.2014.04.009.
cerevisiae yeast and the fermentation method for bioethanol production. Journal of Taheri-Garavand, A., Fatahi, S., Banan, A., Makino, Y., 2019. Real-time nondestructive
Food Process Engineering 42 (7), e13283. https://doi.org/10.1111/jfpe.13283. monitoring of Common Carp Fish freshness using robust vision-based intelligent
Jarolmasjed, S., Espinoza, C.Z., Sankaran, S., Khot, L.R., 2016. Postharvest bitter pit modeling approaches. Computers and Electronics in Agriculture 159, 16–27. https://
detection and progression evaluation in ‘Honeycrisp’apples using computed tomo- doi.org/10.1016/j.compag.2019.02.023.
graphy images. Postharvest Biology and Technology 118, 35–42. https://doi.org/10. Teimouri, N., Omid, M., Mollazade, K., Rajabipour, A., 2016. An Artificial Neural
1016/j.postharvbio.2016.03.014. Network Based Method to Identify Five Classes of Almond According to Visual
Khojastehnazhand, M., Mohammadi, V., Minaei, S., 2019. Maturity detection and volume Features. Journal of Food Process Engineering 39 (6), 625–635. https://doi.org/10.
estimation of apricot using image processing technique. Scientia Horticulturae 251, 1111/jfpe.12255.
247–251. https://doi.org/10.1016/j.scienta.2019.03.033. Teimouri, N., Omid, M., Mollazade, K., Mousazadeh, H., Alimardani, R., Karstoft, H.,
Larada, J.I., Pojas, G.J., Ferrer, L.V.V., 2018. Postharvest classification of banana (Musa 2018. On-line separation and sorting of chicken portions using a robust vision-based
acuminata) using tier-based machine learning. Postharvest Biology and Technology intelligent modelling approach. Biosystems Engineering 167, 8–20. https://doi.org/
145, 93–100. https://doi.org/10.1016/j.postharvbio.2018.06.004. 10.1016/j.biosystemseng.2017.12.009.
Liming, X., Yanchao, Z., 2010. Automated strawberry grading system based on image Utai, K., Nagle, M., Hämmerle, S., Spreer, W., Mahayothee, B., Müller, J., 2019. Mass
processing. Computers and Electronics in Agriculture 71, S32–S39. https://doi.org/ estimation of mango fruit (Mangifera indica L., cv.‘Nam Dokmai’) by linking image
10.1016/j.compag.2009.09.013. processing and artificial neural network. Engineering in Agriculture, Environment
Marti, P., Shiri, J., Duran-Ros, M., Arbat, G., De Cartagena, F.R., Puig-Bargués, J., 2013. and Food 12 (1), 103–110. https://doi.org/10.1016/j.eaef.2018.10.003.
Artificial neural networks vs. gene expression programming for estimating outlet Vélez-Rivera, N., Blasco, J., Chanona-Pérez, J., Calderón-Domínguez, G., de Jesús Perea-
dissolved oxygen in micro-irrigation sand filters fed with effluents. Computers and Flores, M., Arzate-Vázquez, I., Farrera-Rebollo, R., 2014. Computer vision system
Electronics in Agriculture 99, 176–185. https://doi.org/10.1016/j.compag.2013.08. applied to classification of “Manila” mangoes during ripening process. Food and
016. Bioprocess Technology 7 (4), 1183–1194. https://doi.org/10.1007/s11947-013-
Mizushima, A., Lu, R., 2013. An image segmentation method for apple sorting and 1142-4.
grading using support vector machine and Otsu’s method. Computers and Electronics Wan, P., Toudeshki, A., Tan, H., Ehsani, R., 2018. A methodology for fresh tomato ma-
in Agriculture 94, 29–37. https://doi.org/10.1016/j.compag.2013.02.009. turity detection using computer vision. Computers and Electronics in Agriculture
Mohammadi, V., Minaei, S., 2019. Artificial Intelligence in the Production Process. 146, 43–50. https://doi.org/10.1016/j.compag.2018.01.011.
Engineering Tools in the Beverage Industry. Woodhead Publishing, pp. 27–63. Wu, W., Li, A.D., He, X.H., Ma, R., Liu, H.B., Lv, J.K., 2018. A comparison of support
https://doi.org/10.1016/B978-0-12-815258-4.00002-0. vector machines, artificial neural network and classification tree for identifying soil
Mohammadi, V., Kheiralipour, K., Ghasemi-Varnamkhasti, M., 2015. Detecting maturity texture classes in southwest China. Computers and Electronics in Agriculture 144,
of persimmon fruit based on image processing technique. Scientia Horticulturae 184, 86–93. https://doi.org/10.1016/j.compag.2017.11.037.

12