ECOINF-00520; No of Pages 8

Ecological Informatics xxx (2014) xxx–xxx

Contents lists available at ScienceDirect

Ecological Informatics
journal homepage: www.elsevier.com/locate/ecolinf

Automatic identification and counting of small size pests in greenhouse conditions with
low computational cost
Chunlei Xia a,b, Tae-Soo Chon c, Zongming Ren d, Jang-Myung Lee b,⁎
a
The Research Center for Coastal Environmental Engineering and Technology of Shandong Province, Yantai Institute of Coastal Zone Research, Chinese Academy of Sciences, Yantai 264003,
P.R. China
b
School of Electrical Engineering, Pusan National University, Busan (Pusan) 609–735, Republic of Korea
c
Department of Biological Sciences, Pusan National University, Busan (Pusan) 609–735, Republic of Korea
d
College of Life Science, Shandong Normal University, Jinan, 250014, P. R. China

a r t i c l e i n f o a b s t r a c t

Article history: We propose an automatic pest identification method suitable for large scale, long term monitoring for mobile or
Received 31 March 2014 embedded devices in situ with less computational cost. A procedure of segmentation and image separation was
Received in revised form 12 September 2014 devised to identify common greenhouse pests, whiteflies, aphid and thrips. Initially, the watershed algorithm
Accepted 15 September 2014
was used to segment insects from the background (i.e., sticky trap) images. Color feature of the insects were sub-
Available online xxxx
sequently extracted by Mahalanobis distance for identification of pest species. Accuracy and computational costs
Keywords:
were evaluated across different image resolutions. The correlation of determination (R2) between the proposed
Pest monitoring identification scheme and manual identification were high, showing 0.934 for whitefly, 0.925 for thrips, and
Computational complexity 0.945 for aphids even with low resolution images. Comparing with the conventional methods, pests were effi-
Mahalanobis distance ciently identified with low computational cost. Optimal image resolution for species identification regarding
Greenhouse management long-term survey was discussed in practical aspect with less computational complexity.
© 2014 Elsevier B.V. All rights reserved.

1. Introduction visual information (e.g., pest shape, color), it would not produce any
extra disturbances to environment including chemical/physical pollu-
Pest is one of the critical factors causing economic loss in greenhouse tions and could not raise degradation issues to the ecosystems. The eco-
where crops are cultivated in congested conditions in limited areas. In- logical intensification could be achieved by employing the automatic
tegrated Pest Management (IPM) has been widely applied to the agri- pest identification in the agricultural practices that maximizes the pro-
cultural practices to achieve minimizing crop damage, environmental duction while minimizing anthropogenic environmental impacts con-
contamination, and economic loss concurrently (Allen and Rajotte, currently (Bommarco et al., 2013).
1990). One of prerequisites for IPM, however, is to accurately investi- Since the last decade computer hardware and imaging devices sig-
gate population densities of pest species. Objective identification of nificantly contributed to automatic identification of biological organ-
pest species and density estimation is essential for initiating any pest isms (Gaston and O'Neill, 2004; MacLeod et al., 2010). Detection of
management program (Qiao et al., 2008). One of the most common agricultural pests has garnered special attention, especially greenhouse
methods for pest detection in greenhouse, however, has been mainly pests such as whitefly (Bemisia tabaci Genn), aphids (Aphis gossypii
based on conventional sticky traps (Pinto-Zevallos and Vänninen, Glover) and thrips (Thrips tabaci L.). These greenhouse pests are small
2013). Counting the number of insects on sticky traps has been conven- in size and difficult to recognize, but are critical in causing damage
tionally relied on visual judgment by humans (Wise et al., 2007). Due to under congested cultivation conditions. Martin and Thonnat (2007)
complexity of insect morphology automatic identification has been presented a cognitive vision approach that adjusts optimal parameters
considered as a difficult task. Especially with the small size pests, such for segmenting whitefly out of leaves based on adaptive learning tech-
as greenhouse insect pests, the efficiency of human counting is low niques. By employing machine vision and knowledge-base techniques,
and unreliable depending on observation conditions of observers Boissard et al. (2008) proposed a multidisciplinary cognitive vision ap-
(e.g., identification skill, fatigue). Therefore, implementation of auto- proach applicable to whitefly detection in rose in situ. Solis Sánchez
matic pest identification is vital to the modern agricultural production. et al. (2009) utilized the geometric features (e.g., eccentricity, area
Since the automatic pest identification system is mainly based on the size) to whitefly scouting by segmenting the insects from sticky trap
images. Xia et al. (2012) developed a multifractal dimension to detect
⁎ Corresponding author. Tel.: +82 51 510 2378. whitefly in situ, which was robust to field noise such as unequal illumi-
E-mail address: [email protected] (J.-M. Lee). nation changes and light reflections on trap surface.

http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
1574-9541/© 2014 Elsevier B.V. All rights reserved.

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
2 C. Xia et al. / Ecological Informatics xxx (2014) xxx–xxx

Previous automatic identification, however, was mainly conducted insects ranged in 13–152 individuals, 13–20 individuals and 27–83 individ-
with one species, whiteflies. Other common species such as aphids uals for whitefly, aphids, and thrips, respectively. Specifically the average
and thrips are also important in greenhouse conditions but were not ex- was 42.0 ± 39.7 individuals per trap for whitefly, 17.2 ± 8.8 individuals
tensively studied with image processing. Cho et al. (2007) proposed an per trap for aphids, and 59.8 ± 20.4 individuals per trap for thrips. To
automatic pest identification method based on YUV color space and build the classifier for identification of three pest species, twenty samples
fixed thresholds to examine pests from the sticky traps collected in for each pest species were randomly selected from the rest of 5 sticky traps.
greenhouse. Color components and body size of pest were utilized to For segmentation watershed was selected since the algorithm has
identify whitefly, aphids and thrips in this study, but identification been widely used in image processing including object detection
had to be conducted with high resolution images (600 DPI) only. (Boissard et al., 2008) and insect (i.e., whiteflies) body segmentation
Gabor filter and histogram of gradients were additionally utilized to ex- from field images (Xia et al., 2012). Direct application of the watershed
tract the feature of pests in order to identify whitefly and greenfly from segmentation, however, produces a number of unexpected regions in
sticky trap in field conditions by Kumar et al. (2010), and they applied addition to the segmentation targets (Ng et al., 2006). In order to
the support vector machine (SVM) to identification of the pests. Solis- solve this problem, the marker controlled watershed has been intro-
Sánchez et al. (2011) utilized scale invariant feature transform (SIFT) duced as well to decrease the sensitivity to noise and to enhance the re-
as a feature descriptor of the segmented insect images, combining liability of watershed segmentation (Parvati et al., 2008).
with shape feature of insect body. These methods, however, in general Mahalanobis distance has been reported to be effective in measuring
also require either high resolution of images (Cho et al., 2007; the similarity of a set of given values to a set of values obtained from the
Solis-Sánchez et al., 2011) or computationally expensive recognition sampled data. Mahalanobis distance was successful in image recogni-
algorithms (e.g., Gabor filter, SIFT). Although high pest recognition tion or segmentation (e.g., character recognition, plant segmentation,
accuracy was achieved by these methods, the methods are in practical plant disease identification) (Kato et al., 1999; Manh et al., 2001;
aspect not specifically suitable for large scale, long term pest monitoring Weinberger et al., 2006; Xia et al., 2013). In order to measure the simi-
(e.g., agricultural sensing network) in field conditions due to high com- larity between given specimens to the three species, whitefly, aphids
putational cost. Since portable devices or wireless sensor network are and thrips, variation of the body color was obtained from the insect
usually powered by small capacity batteries, the high computational samples. The Mahalanobis distance is defined as (De Maesschalck
cost would be prone to a high risk of power loss for data processing. et al., 2000):
An alternative solution may be to transmit the field images to the
remote terminal (e.g., personal computer) for pest identification. But qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
the transmission ability of the wireless network would be still a D¼ ðx−μ ÞT S−1 ðx−μ Þ ð1Þ
constraining factor since high resolution images would require a broad-
band network of which the transmission range is limited, especially in
the large scale agricultural field (Koumpouros et al., 2004). Although re- where μ is the mean color of the insects, S− 1 represents the inverse
cent 3G/4G mobile network may provide a technical solution to the of covariance matrix (S) of the color variations of the sampled
transmission of images in large scale (i.e., data transmission between insects, T indicates the transpose operation, and x is the input
cites), the cost of the mobile network would not be affordable for agri- feature vector of given specimen to be classified. Mean colors and
cultural practices by dealing with high resolution data. Effective pest covariance matrices of each specimen image were obtained from
recognition algorithm with low computation cost is desired for large the insect samples.
scale pest monitoring in practical aspect. Species recognition was conducted by images across different reso-
In this paper, we propose a procedure of image processing suitable lutions. Different level of resolution was controlled by adjusting pixel
for species identification with low resolution images of small size in- numbers. The original image resolution acquired from the scanner was
sects by combining methods for image segmentation and separation. 3500 × 2350 pixels (600 DPI), and converted to lower resolutions of
A classical segmentation algorithm, marker controlled watershed, 1600 × 1100, 1024 × 690, 800 × 540, and 640 × 430 pixels. The pest
was initially applied to the YCbCr color space of sticky trap image. identification program was developed in Matlab 2011b (Mathworks)
Subsequently Mahalanobis distance was used as the classification crite- on a personal computer equipped with Intel G860® CPU with 4GB
rion for species identification by differentiating the color components RAM memory.
and insect size. With less computational complexity based on these
algorithms, efficient identification was achieved with reduction in com- 3. Procedure
putation time, while still maintaining high precision rate for identifica-
tion of three major pests in greenhouse. 3.1. Overall process

2. Materials and methods The overall process of pest identification is presented in Fig. 1. First, in-
dividual insect images were segmented from the sticky trap on the YCbCr
Three species in greenhouse, whitefly (Bemisia tabaci Genn), aphids color spaces. The marker controlled watershed segmentation was utilized
(Aphis gossypii Glover) and thrips (Thrips tabaci L.), which reported to to extract each individual insect from Cb component of YCbCr color spaces.
cause serious damage in greenhouse crops over the world (Malais and Candidate insect images were determined by examining the area size of
Ravensberg, 2004), were selected as the target species for identification the segmented blobs. Each individual insect was represented by mean
in this study. The sticky traps (15 cm × 10 cm) were placed 10 cm above color of the insect body in HSV color space. Mahalanobis distance was
tomato crops that were approximately 160 cm tall. The borders of the then introduced as a classifier for identifying species.
sticky trap were encapsulated by cardboard with 2 mm thickness. In Mean colors and covariance matrices of each specimen image were
total, fifteen sticky traps were selected for collecting species for one obtained from the sample specimen. In the identification process,
week. Methods for plant cultivation and insect observation were report- Mahalanobis distances between the given image of insect and target
ed in (Chung et al., 2014; Qiao et al., 2008). After field collection, sticky species were calculated based on the obtained mean colors and covari-
trap images were acquired by the HP G3110® scanner. Ten of the sticky ance matrices. The specimen was identified to the species showing the
traps were randomly selected for developing the species identification nearest Mahalanobis distance. If the nearest Mahalanobis distance was
program. The total number of pests attached on the ten sticky traps longer than the predefined threshold which had been estimated from
was ranged between 62 and 252 individuals per trap with the average pretests, the candidate image was regarded as noise and was removed
number of pests of 115.2 ± 53.9 individuals per trap. The number of from the identification list.

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
 C. Xia et al. / Ecological Informatics xxx (2014) xxx–xxx 3

3.2. Insect segmentation

The sticky trap images mainly contain 5 colors of objects including

yellow background, green grids, and characteristic tones of black
(Aphids), white (Whitefly), and yellowish brown (Thrips) (Fig. 2a and
b). Green color grids (and alphabets) printed on the yellow sticky trap
was a strong source of heterogeneity against the yellow background of
the sticky trap in the image. Indeed green grids caused severe noises
in insect segmentation (Fig. 2a and c). Since the color of yellow, white
and green has similar RGB values, it was difficult to accurately separate
images of three species (Cho et al., 2007). In order to effectively distin-
guish the insects from grids and background, the sticky trap images
need to be converted from RGB to other color spaces. In this study,
three species were segmented in YCbCr color space. Yellow background
and green grids can be clearly distinguished from insects based on the
Cb component of YCbCr color space. As shown in Fig. 2(c), intensity
values of insects on Cb component matched well to positions of individ-
uals (Fig. 2a) and were much higher than the values of background and
green grids. Consequently, the green grids, regarded as noise in this
study, can be effectively removed from the sticky trap images.
The markers for conducting the watershed segmentation were ob-
tained by calculating the regional minima and maxima. Watershed
transform was then performed based on the regional maxima.
Fig. 3(a) shows the segmented candidates of insects (Fig. 2a and
c) with white blobs. Inaccurate segments, however, are also included
within the insect candidates. After segmentation, the candidate images
were determined by examining the area size of the segmented blobs
produced by the watershed. The segmented blobs with improper size
were further eliminated; green letters on the sticky trap (Fig. 2a) or
local color distortion on the background of sticky trap usually produced
large sized blobs.
According to the image resolution, appropriate range of body size
was provided as a threshold to remove the unexpected large or small
Fig. 1. Schematic flow of insect segmentation and species identification.
segments. For example, body area of insect was ranged in 2 to 60 pixels

Fig. 2. Images of sticky trap and samples. (a) Trap, (b) three insect image recorded on the trap, and (c) Cb component of sticky trap image.

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
4 C. Xia et al. / Ecological Informatics xxx (2014) xxx–xxx

Fig. 3. Insect segmentation by watershed algorithm. (a) Candidate insect images, and (b) identified insects marked by red crosses. Notice that the large-size blobs were originated from the
green letters.

with image resolution of 800 × 540 pixels. The ranges of insect accurately identified based on size. However, three species, whitefly,
area were mostly determined according to pretests across different aphids and thrips, showed different colors (Fig. 2b); the color features
resolutions. The determined insects were marked with red crosses as of these insects was in fact a critical factor for identifying pest species
shown in Fig. 3. The unexpected large blobs were successfully removed in this study.
from the segmentation results. The mean color μ and covariance matrix S for Mahalanobis distance
could be obtained from insect samples in different species. For the seg-
3.3. Species identification mented insects (Fig. 4), mean color of the insect body was calculated as
the input vector x. Three Mahalanobis distances were obtained for the
After segmentation, images of individual insects are separated and input vector x (of input specimen) to each of three species. By compar-
extracted from the background (Fig. 4). The segmented blobs were ing the measured distances, the insect was classified to species showing
numbered as shown in the left part of Fig. 4. The images of individuals the nearest distance. In this study, color vector x was represented by
were accordingly extracted and presented in the right panel of Fig. 4. HSV color instead of RGB and YCbCr. Although the insects were accurate-
Low image resolution (800 × 540 pixels) was presented in this case: ly segmented in the YCbCr color space, the intensity value of these insect
the insect body size was less than 20 pixels. Therefore, the contours of on Cb component were similar and difficult to distinguish three species
the insects were not smooth and shapes of the insect could not be from these images. HSV color representation, however, demonstrated
high performance in pest identification in the pre-tests. In addition, a
predefined threshold value was given to examine the validation of the
classification results. Species identification would be invalid if the
nearest Mahalanobis distance is longer than the threshold. Since
incorrectly-segmented blobs could be produced during the course of
the segmentation process and could cause additional false alarms,
noises were further removed by thresholding. The threshold was esti-
mated from the preliminary tests; the value of 80 was used for all
image resolutions.

4. Experimental results

Pest identification experiments were conducted in two steps: seg-

mentation of individual insects from the sticky trap images and species
identification of candidate images as stated above. Initially, detection of
insect images was carried out by using the marker controlled watershed
Fig. 4. Extraction of individual insects (Numbers indicating the sequence of segmented algorithm (see Section 3.2). The accuracy of insect segmentation and
blobs). computational time were evaluated across different levels of resolution.

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
 C. Xia et al. / Ecological Informatics xxx (2014) xxx–xxx 5

The precision and recall analysis were applied to determining ac-

curacy of insect detection (Xia et al., 2012). Precision is defined as
the proportion of correct segmentation of insect (true positives) to
the total segmentation results and is expressed as True positive /
(True positive + False positive). Recall is measured as True positive /
(True positive + False negative). The false positive could be also re-
ferred as false alarm since the non-insect (e.g., noise) is reported as an
insect. In addition the false negative would stand for miss-detection of
insects. The F value based on precision and recall analysis represents
the overall performance of the insect detection including miss detection
and false alarms:

2  Precision  Recall
F¼ : ð2Þ
Precision þ Recall

High F value indicates more correct detection with fewer false

alarms at the same time. As presented in Fig. 5, the F value varied
from 0.96 to 0.92; overall performance across different image resolu-
tions was in general over 0.90. The highest F value, 0.96, was obtained
from the 800-pixel resolution, followed by 0.95 with the 640-pixel res-
olution. The detection accuracy on 640-pixel resolution was only slight-
ly lower than the best detection accuracy (800-pixel resolution). On the
contrary, the detection accuracy tended to decrease as the image resolu-
tion increased higher than 800-pixel although the degree was slight.
The images with the highest resolution that provided detailed informa-
tion of insects showed lower detection accuracy (F) in fact, although
requiring much higher computational cost (Fig. 5). Due to over-
segmentation in the higher resolution images, false alarms occurred fre-
quently. Especially, the slight color changes on the yellow background
could be easily confused with whitefly. The standard deviation was Fig. 6. Variation and computational cost (s). (a) Standard deviation, and (b) cost propor-
tion (%) of specific identification processes, including classification, blob processing and
also presented as error bars with precision (F) in Fig. 5. The highest stan-
segmentation, in relation to the total time cost across different levels of image resolution.
dard deviation was found at the highest image resolution which indicat-
ed the segmentation performance with resolution of 3500-pixel was
less robust to noises. processing includes morphological processing (e.g., erosion and
The computational time increased greatly with the increase in image dilation) and removal of the blob with unsuitable size (i.e., too large or
resolution (Fig. 5). Especially, the image processing time with the too small sized blobs). Due to over-segmentation problems occurred
highest resolution (3500-pixel, 600 DPI) increased 4 times (32.3 s) com- with the high quality images, more blobs were produced by the seg-
paring with the processing time with the 1600-pixel resolution (6.36 s). mentation processing. In addition to the image segmentation, blob pro-
This was 37 times longer than the processing time by the 640-pixel res- cessing on the high quality images cost much higher in dealing with the
olution (0.87 s). Since the computational time showed very large differ- segmented blobs, comparing with the processing time with the low
ences from the low to the high image resolution, the standard deviation quality images. The computational time for both segmentation and
was too small to plot in the line chart (Fig. 5). Instead, the standard de- blob processing dramatically increased by more than 20 times from
viation is separately provided in Fig. 6(a), showing higher values with 640-pixel images (1.06 s) to 3500-pixel images (34.51 s)(Fig. 5). This
increase in image resolution. It was noteworthy that standard deviation is understandable since the number of pixels increased 30 times from
was extremely high with the maximum resolution, 3500-pixel. 640-pixel to 3500-pixel in resolution. The time for identification also
The source of time consumption for classification, blob processing, increased substantially about 11 times at the same image resolution.
and segmentation is specifically presented in Fig. 6(b). The blob On the contrary, the proportion of the classification time in relation to
the total time cost decreased from 19.0% to 6.4% in images with maximal
resolution (Fig. 6b). The computational cost of the Mahalanobis
classifier was rather low, even the image resolution increased greatly.
In fact, the increase of classification time was mainly caused by color
feature extraction, since the insect image size geometrically increased
with increase in pixels.
To evaluate pest identification accuracy the number of individuals
for each species was separately counted by humans with experience
in insect identification. It was noteworthy that the humans needed 5
to 20 minutes to examine the pests collected from one sticky trap.
Maximal identification time was 40.6 seconds on the highest image res-
olution (3500-pixel). The average time consumptions by the automatic
identification, however, were only 1.06 seconds on 640-pixel, 1.33 -
seconds for 800-pixel and 2.35 seconds for 1024-pixels.
The accuracy of species identification was evaluated by correlation
of determination (R2) between the automatic identification and
human counting (Table 1). The overall highest accuracy was achieved
Fig. 5. Computational cost (s) and precision (F) across different levels of image resolution with whitefly maintaining maximal R2 ranging 0.927 - 0.957 across dif-
(Error bars indicate standard deviation of precision). ferent image resolutions (Table 1). Aphid showed the maximum

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
6 C. Xia et al. / Ecological Informatics xxx (2014) xxx–xxx

Table 1
Coefficient of determination (R2) in pest identification between the proposed method and human counting of three insects across different image resolutions (SD indicates standard
deviation).

Insect Image resolution (pixel) Average ± SD

640 800 1024 1600 3500

Aphids 0.881 0.934 0.941 0.912 0.909 0.915 ± 0.023

Thrips 0.916 0.925 0.853 0.692 0.508 0.779 ± 0.093
Whitefly 0.951 0.945 0.957 0.937 0.927 0.943 ± 0.007
Average ± SD 0.916 ± 0.029 0.935 ± 0.008 0.917 ± 0.046 0.847 ± 0.110 0.781 ± 0.193

recognition rate across different image resolutions, comparing with whitefly and thrips. For instance the thrips were confused with whitefly,
other species. The highest identification accuracy (0.941) was obtained and vice versa, at maximum resolution, showing the error rates as 0.144
from image resolution of 1024-pixel for aphid. The recognition rate of and 0.087, respectively, whereas the values were substantially lower at
thrip was high with image resolutions of 640, 800 and 1024 pixels, other resolutions (Fig. 7). Whitefly and thrips tended to be easily con-
showing correlation of determination (R2) as 0.916, 0.925 and 0.853, fused in the classification results. Caution is needed in identifying
respectively. It was worth noting that the correlation of determination aphids. Segmentation of aphids occasionally ended up with several
(R2) was only 0.692 and 0.508 with the high image quality (1600 and small segments, some of which were recognized as single individual
3500 pixels images) for thrips. This was mainly due to false alarms. although the segment is only a part of the body.
Identification accuracy of thrips considerably decreased when image By taking into account the correlation of determination, precision
resolution was higher than 800 pixels. The overall identification perfor- analysis, and the computational cost all together (Tables 1 and 2, and
mance for three species was superior with image resolution of Fig. 7), the image resolution of 800-pixel could be an optimal choice
800 pixels (0.935), followed by 1024 pixels (0.917) and 640 pixels for detecting and counting the three major pests in greenhouse. The
(0.916) (Table 1). image resolution of 640-pixel could be also an option to detect pests
Precision analysis (Table 2) was further performed to examine the in practical aspect, considering low computational cost.
correctness achieved by identification results; a high precision value in-
dicates that the identification results contain a high percentage of cor- 5. Discussion
rectly identified pests and a low percentage of false alarms as well
(Xia et al., 2012), as stated above. False alarms frequently occurred in Cho et al. (2007) reported the segmentation of three species, aphids,
the identification results, affecting overall identification accuracy. The whitefly and thrips, by two different approaches: aphids were detected
precision analysis reveals the reliability of the identification results. As in RGB image whereas whitefly and thrips were segmented in YUV color
shown in Table 2, the precision values are comparatively higher in the space. In this study, three species of pests were segmented based on one
low image resolutions. Precision was 0.905 for aphid, 0.955 for thrip method for concurrent identification of three species from the sticky
and 0.871 for whitefly with image resolution of 640 pixels whereas trap images. The preliminary experiments of insect segmentation
the values were 0.941, 0.869 and 0.870 with 800 pixels, respectively. were conducted on widely used color spaces including CIELab, HSV
The precision was in general lower in the higher resolution of images and YCbCr color models (Payne et al., 2014; Sural et al., 2002; Zheng
(N 1024 pixels). The precision of identifying thrips and whitefly even and Lu, 2012). The YCbCr color model showed the better segmentation
decreased significantly to 0.692 and 0.654, respectively, with maximum comparing with CIELab and HSV. In our study the insects and grids on
image resolution of 3500 pixels (Table 2). Since over-segmentation in- the sticky traps were presented as local maxima in the Cb channel of
creased in high resolution images, false alarms correspondingly in- YCbCr color model (Fig. 2c). Consequently the local maxima on the
creased to induce reduction in precision. Inaccurate segmentation, image could be extracted as the markers for the marker-controlled wa-
which often contained the surrounded background in the insect images, tershed segmentation. The green grids, however, showed much lower
also caused the inaccurate mean color of insects for identifying pest spe- intensity values in the Cb channel than common greenhouse species
cies. Aphids overall showed the best recognition performance (R2) (Fig. 2c). Therefore, the accuracy of insect segmentation was higher in
(Table 2) with different image resolutions although the maximum pre- the Cb channel. The segmentation from the yellow sticky traps was
cision was achieved with whitefly (Table 1). Due to distinctive size also demonstrated with the green color insects such as greenfly
(i.e., large) and color (i.e., black), aphids overall showed higher feasibil- (Kumar et al., 2010), however the segmentation in this case was only
ity in identification comparing with other species. conducted with the traps without printed green grids and alphabets.
The confusion matrix presents the degree of correct answer along We demonstrated that segmentation was even possible with images
with the identification errors for the three insects (Fig. 7) (Kohavi and showing grids and alphabets in this study (Fig. 2a).
Provost, 1998). The diagonal elements of the confusion matrix indicate Although multifractal demonstrated showed higher performance re-
the true positives. The rest of elements in rows represent the false pos- garding accuracy than watershed on insect segmentation against noise
itives of classification. Overall diagonal elements showed the maximum in field conditions (e.g., illumination reflections) (Xia et al., 2012),
values as expected. Substantially small numbers were found at either multifractal is much more computationally expensive. Consequently,
row or column elements. The values decreased in general along with in- automation with multifractal would not be affordable for long term op-
crease in image resolution, especially at maximum resolution for eration of portable devices. Otsu algorithm (Otsu, 1979) demands less

Table 2
Precision in pest identification of three insects across different image resolutions (SD indicates standard deviation).

Insects Image resolution (pixel) Average ± SD

640 800 1024 1600 3500

Aphids 0.905 0.941 0.858 0.828 0.746 0.856 ± 0.043

Thrips 0.955 0.869 0.790 0.806 0.692 0.822 ± 0.065
Whitefly 0.871 0.870 0.818 0.773 0.654 0.797 ± 0.041
Average ± SD 0.910 ± 0.034 0.893 ± 0.034 0.822 ± 0.028 0.802 ± 0.023 0.697 ± 0.038

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
 C. Xia et al. / Ecological Informatics xxx (2014) xxx–xxx 7

Fig. 8. Images of small size pests difficult for recognition. (a) Over-segmentation of large
pests, and (b) ambiguous objects for identification.

thrips were occasionally divided into two parts in the high resolution
images (Tables 1 and 2). With low resolution images on the other
hand, image feature of legs and wings were not strong and were
neglected by the segmentation algorithm. The optimal level of resolu-
tion was required as shown in this study.
In order to deal with the high quality images, more sophisticated
algorithms would be necessary such as scale invariant feature transfor-
mation (SIFT) (Solis-Sánchez et al., 2011). However, since the appear-
ance of three species (thrips, aphids and whitefly) were mainly
Fig. 7. Confusion matrices for human counting and predication in recognition of three in- distinguishable with color features in field conditions, the complicated
sects across different image resolutions (Darker color indicates higher accuracy for diago-
algorithms applied to high quality images were not necessary to detect
nal elements).
and identify pests collected on sticky traps in field conditions in practi-
cal aspect as demonstrated in this study.
computational cost, however the method is mainly designed for global Although aphids showed the best recognition performance, inaccu-
thresholding and is highly sensitive to illumination changes (Xia et al., rately segmented aphids led to incorrect identifications. Some segmented
2012). The local adaptive thresholding (Kom et al., 2007) could be an- images of aphids contained their legs or the surrounding backgrounds,
other option to extract the insects from sticky trap images without therefore, the mean color of the aphid was sometimes distorted since
green grids as reported by Kumar et al. (2010) and Bechar et al. the legs and background color were included in calculating the mean
(2010). In this case, however, the green grids on the sticky traps should color. The aphids could be miss-recognized as thrips in these cases. The
be separately segmented before identifying insects. Considering extrac- incorrect identification of aphids, however, was not frequent, especially
tion of the green grids, the watershed (Vincent and Soille, 1991) could with the low image resolution.
be an optimal solution to acquire accurate segmentation of insects Identification performance of thrips was relatively lower comparing
from backgrounds in acceptable precision level while achieving lower with other species, since small size thrips are often confused with
computational cost concurrently. Computational complexity of insect whitefly as presented in the confusion matrices (Fig. 7). As presented
segmentation could be further reduced by the integrating adaptive in Fig. 8(b), the pest in the center was identified as whitefly and marked
thresholding algorithms (Rahman and Islam, 2013). This will be investi- with a red letter ‘W’. The mean color of that pest was close to whitefly
gated in the future study. but the pest body was in fact surrounded by gray shadow, which also
Segmentation accuracy showed a slightly decreasing trend as fit to the feature of thrips. These pests were incorrectly identified as
resolution increased in this study (Fig. 5). This was due to hyper sensi- whitefly in this case as shown in Fig. 8(b). In addition, the overlapped
tivity to noise caused by high image resolution. Although high quality image of specimens may also reduce the identification accuracy. The
images (N1600 pixels) contained details of the pests, the detection pest size was rather smaller in this study, and there was a low possibility
results turned out to be too sensitive to environmental effects of overlapping. In the future, however, new method should be devised
(e.g., illumination change). Details of image were amplified in high res- in case pest population size increased and overlapping of specimens
olution and more local maxima were produced while extracting pest would increase on the sticky trap.
images from backgrounds. Since the watershed segmentation was Light condition was an additional problem for insect identification.
based on the local maxima, the large number of local maxima led to pro- Illumination conditions vary in a high degree in natural condition and
duction of more watershed segmentations, causing hypersensitivity to would cause color distortion in image capturing in greenhouse especial-
noise subsequently. The inaccurate segmentation or partially segment- ly with high resolution images. In this study, it was aimed to present a
ed insect images were an important source of the identification errors prototype of automatic pest identification system, and the current
according to authors' experience. Legs and wings of the large aphids, version of the proposed method still needs further implementation
for example, were occasionally segmented as separate objects to serve tests in response to various environmental factors (e.g., continuously
as extra noise sources. The ill-segmented objects were sometimes in- changing illumination conditions) and different population size to
correctly identified as different species (e.g., thrips) to make a false sig- build the products for satisfactory use in agricultural practices.
nal (Fig. 8a). Since thrips' body contains gray and white areas together, Color correction (Rizzi et al., 2003) or automatic parameter tuning

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006
8 C. Xia et al. / Ecological Informatics xxx (2014) xxx–xxx

(Boissard et al., 2008) should be also considered in this regard in the Kumar, R., Martin, V., Moisan, S., 2010. Robust insect classification applied to real time
greenhouse infestation monitoring. Proceedings of the 20th International Conference
future. In addition, real scale field tests would be further needed to eval- on Pattern Recognition on Visual Observation and Analysis of Animal and Insect
uate identification performance over space (e.g., whole green house) Behavior Workshop, pp. 1–4.
and time (e.g., during whole procedure of cultivation). Further study is MacLeod, N., Benfield, M., Culverhouse, P., 2010. Time to automate identification. Nature
467 (7312), 154–155.
required in this regard with the aid of field assessment methods and Malais, M.H., Ravensberg, W.J., 2004. Knowing and Recognizing: The Biology of Glass-
integrative pest management programs. house Pests and Their Natural Enemies. Koppert BV, The Netherlands.
Manh, A.G., Rabatel, G., Assemat, L., Aldon, M.J., 2001. AE—automation and emerging
technologies: weed leaf image segmentation by deformable templates. J. Agric. Eng.
6. Conclusion Res. 80 (2), 139–146.
Martin, V., Thonnat, M., 2007. A cognitive vision approach to image segmentation.
By using watershed algorithm and Mahalanobis distance on YCrCb Proceedings of the 19th IEEE International Conference on Tools with Artificial Intelli-
gence Washington, DC, USA, pp. 265–294.
color space, a pest identification procedure was proposed, achieving Ng, H.P., Ong, S.H., Foong, K.W.C., Goh, P.S., Nowinski, W.L., 2006. Medical image segmen-
both low computational cost and high feasibility in identifying three tation using K-means clustering and improved watershed algorithm. IEEE Southwest
common species, aphid, thrips and whitefly, in greenhouse. The pest Symposium on Image Analysis and Interpretation, pp. 61–65.
Otsu, N., 1979. A threshold selection method from gray-level histograms. Automatica 11
detection and species identification accuracy were verified through (285–296), 23–27.
experiments with specimens collected on sticky traps in greenhouse Parvati, K., Rao, P., Mariya Das, M., 2008. Image segmentation using gray-scale morphol-
conditions. The proposed method demonstrated high performance in ogy and marker-controlled watershed transformation. Discret. Dyn. Nat. Soc. 8
(Article ID 384346).
identifying and counting small sized pests especially with low resolu-
Payne, A., Walsh, K., Subedi, P., Jarvis, D., 2014. Estimating mango crop yield using image
tion images (i.e., 800-pixel, 640-pixel) when compared with human analysis using fruit at ‘stone hardening’ stage and night time imaging. Comput.
identification results. The proposed method would be especially practi- Electron. Agric. 100, 160–167.
Pinto-Zevallos, D.M., Vänninen, I., 2013. Yellow sticky traps for decision-making in white-
cal for large scale, long term monitoring by using the portable devices
fly management: what has been achieved? Crop. Prot. 47, 74–84.
which require minimal power consumption and endurable data storage. Qiao, M., Lim, J., Ji, C.W., Chung, B.K., Kim, H.Y., Uhm, K.B., Myung, C.S., Cho, J., Chon, T.S.,
2008. Density estimation of Bemisia tabaci (Hemiptera: Aleyrodidae) in a greenhouse
using sticky traps in conjunction with an image processing system. J. Asia Pac.
References
Entomol. 11 (1), 25–29.
Rahman, M., Islam, M., 2013. Segmentation of color image using adaptive thresholding
Allen, W.A., Rajotte, E.G., 1990. The changing role of extension entomology in the IPM era.
and masking with watershed algorithm. Proceedings of International Conference on
Annu. Rev. Entomol. 35 (1), 379–397.
Informatics, Electronics & Vision, pp. 1–6.
Bechar, I., Moisan, S., Thonnat, M., Bremond, F., 2010. On-line video recognition and
Rizzi, A., Gatta, C., Marini, D., 2003. A new algorithm for unsupervised global and local
counting of harmful insects. The 20th International Conference on Pattern Recogni-
color correction. Pattern Recogn. Lett. 24 (11), 1663–1677.
tion (ICPR), pp. 4068–4071.
Solis Sánchez, L., García Escalante, J., Castañeda Miranda, R., Torres Pacheco, I., Guevara
Boissard, P., Martin, V., Moisan, S., 2008. A cognitive vision approach to early pest detec-
González, R., 2009. Machine vision algorithm for whiteflies (Bemisia tabaci Genn.)
tion in greenhouse crops. Comput. Electron. Agric. 62 (2), 81–93.
scouting under greenhouse environment. J. Appl. Entomol. 133 (7), 546–552.
Bommarco, R., Kleijn, D., Potts, S.G., 2013. Ecological intensification: harnessing ecosystem
Solis-Sánchez, L.O., Castañeda-Miranda, R., García-Escalante, J.J., Torres-Pacheco, I.,
services for food security. Trends Ecol. Evol. 28 (4), 230–238.
Guevara-González, R.G., Castañeda-Miranda, C.L., Alaniz-Lumbreras, P.D., 2011.
Cho, J., Choi, J., Qiao, M., Ji, C.W., Kim, H.Y., Uhm, K.B., Chon, T.S., 2007. Automatic identi-
Scale invariant feature approach for insect monitoring. Comput. Electron. Agric. 75
fication of whiteflies, aphids and thrips in greenhouse based on image analysis. Int. J.
(1), 92–99.
Math. Comput. Simul. 346 (246), 244.
Sural, S., Qian, G., Pramanik, S., 2002. Segmentation and histogram generation using the
Chung, B.K., Xia, C., Song, Y.H., Lee, J.M., Li, Y., Kim, H., Chon, T.S., 2014. Sampling of
HSV color space for image retrieval. Proceedings of International Conference on
Bemisia tabaci adults using a pre-programmed autonomous pest control robot. J.
Image Processing 2, p. II-589.
Asia Pac. Entomol. 17 (4), 737–743.
Vincent, L., Soille, P., 1991. Watersheds in digital spaces: an efficient algorithm based on
De Maesschalck, R., Jouan-Rimbaud, D., Massart, D.L., 2000. The mahalanobis distance.
immersion simulations. IEEE Trans. Pattern Anal. Mach. Intell. 13 (6), 583–598.
Chemom. Intell. Lab. Syst. 50 (1), 1–18.
Weinberger, K., Blitzer, J., Saul, L., 2006. Distance metric learning for large margin nearest
Gaston, K.J., O'Neill, M.A., 2004. Automated species identification: why not? Philos. Trans.
neighbor classification. Adv. Neural Inf. Process. Syst. 18, 1473.
R. Soc. Lond. B Biol. Sci. 359 (1444), 655–667.
Wise, J.C., Gut, L.J., Isaacs, R., 2007. Michigan Fruit Management Guide. Department of
Kato, N., Suzuki, M., Omachi, S.I., Aso, H., Nemoto, Y., 1999. A handwritten character rec-
Plant Pathology and Department of Horticulture, Michigan State University.
ognition system using directional element feature and asymmetric Mahalanobis dis-
Xia, C., Lee, J.M., Li, Y., Chung, B.K., Chon, T.S., 2012. In situ detection of small-size insect
tance. IEEE Trans. Pattern Anal. Mach. Intell. 21 (3), 258–262.
pests sampled on traps using multifractal analysis. Opt. Eng. 51 (2), 027001-1.
Kohavi, R., Provost, F., 1998. Glossary of terms: special issue on applications of machine
Xia, C., Lee, J.M., Li, Y., Song, Y.H., Chung, B.K., Chon, T.S., 2013. Plant leaf detection using
learning and the knowledge discovery process. Mach. Learn. 30, 271–274.
modified active shape models. Biosyst. Eng. 116 (1), 23–35.
Kom, G., Tiedeu, A., Kom, M., 2007. Automated detection of masses in mammograms by
Zheng, H., Lu, H., 2012. A least-squares support vector machine (LS-SVM) based on fractal
local adaptive thresholding. Comput. Biol. Med. 37 (1), 37–48.
analysis and CIELab parameters for the detection of browning degree on mango
Koumpouros, Y., Mahaman, B.D., Maliappis, M., Passam, H.C., Sideridis, A.B., Zorkadis, V.,
(Mangifera indica L.). Comput. Electron. Agric. 83, 47–51.
2004. Image processing for distance diagnosis in pest management. Comput.
Electron. Agric. 44 (2), 121–131.

Please cite this article as: Xia, C., et al., Automatic identification and counting of small size pests in greenhouse conditions with low computational
cost, Ecological Informatics (2014), http://dx.doi.org/10.1016/j.ecoinf.2014.09.006