Automated Visual Fruit Detection for Harvesting

Automated Visual Fruit Detection for Harvesting

Ritvik Khandelwal1, Shrenik Ganguli1, Aayush Gandhi1, Dakshit Shah1 Dwayne Gonsalves1, Jaimin Shah1, Aarti A.
Ambedkar2
1
U. G. Student, Department of Electronics and Telecommunication, D.J.Sanghvi College of Engineering, Vile Parle (W), Mumbai- 400056
2
Professor & Head,Electronics and Telecommunication Department D.J.Sanghvi College of Engineering, Vile Parle (W), Mumbai- 400056
E-mail: [email protected], [email protected] , [email protected], [email protected],
[email protected], [email protected], [email protected]

Abstract— The quest of learning computer vision and much more food, and ranchers will confront genuine strain
utilizing it for growth of mankind has been a long one. Over to stay aware of interest.
the most recent twenty years different procedures have been The accurate detection of different fruits is considered
proposed. As the quantity of handling methods continue to challenging and essential tasks. For the achievement of
expand, it gets simpler to make innovations which can accurate results at these tasks, the need of a high-quality
without much of a stretch distinguish objects. The quantity labelled dataset is required. However, there is a lack of such
of object locators accessible today indisputably show that publicly available benchmark datasets for precision farming,
identification should be possible with less complex which restricts the efficient application of modern
strategies which are suitable for a large portion of the technologies, like machine learning algorithms, in
clients. This opens the chance of improving the current greenhouses.
object detectors and makes it cost proficient. This paper
proposes the application of current innovations in
cultivating, it presents the detection of a ripened fruit by an II. OBJECTIVE
automated cost-effective Fruit Harvester. This project is The objective of our project is to be able to detect different
intended to help the farmers save their effort by making types of fruits that are required to be harvested or plucked
their work convenient which consequently increases from the tree. The autonomous robot uses computer vision
productivity in order to detect the fruits with the help of a camera
module. This real time detection is aided by the images and
Keywords — OpenCV, Vision recognition system, Fruit other data used to train the algorithm used. Once there is a
Detection machine vision, fruit harvesting robot, YOLO, fruit detected it is scanned to detect if it is the appropriate
RPI Camera module, Image processing colour for harvesting. Once this process is completed the
robotic arm attached to the robot is placed in position to
harvest the fruit from the tree.
I. INTRODUCTION
III. PROPOSED DESIGN
Horticulture assumes a critical part in mankind's regular day
to day existence as well as in the worldwide economy. Due
to the lack of manpower to carry out the harvesting process The project will comprise a camera module that will capture
it has become increasingly necessary to integrate technology images from the reaEl-time video at 30 frames per second.
into this process. Progressed innovations can be utilized to These images would then undergo various pre-processing
give computerized answers for physically performed techniques. The preprocessing will be done on the
assignments supporting, along these lines, accuracy Raspberry pi. Method 1 uses OpenCV. Method 2 uses
cultivating. Thus, the incorporation of those cutting-edge OpenCV and neural networks (Tensorflow/PyTorch).
advancements in cultivating represents a moving
assignment to the exploration local area from an assortment We have used the Open-CV which is an open-source
of logical fields like Remote Sensing, Machine Learning or platform for the overall implementation of the project. It
Robotics. is a powerful framework which has many algorithms and
Today there has been innovation to mechanize most of the techniques for identifying and classifying images. It
human work however it is least in the field of Agriculture. helps to pre-process data and is much faster than most of
The farming area in India endures most considering its old the other libraries with a very high operational speed
techniques for reaping which require heaps of labour and with minimum glitches. It has in-built trained models
results in helpless yields. At the core of this wonder is the which reduce out setup time for training models for
requirement for essentially expanded creation yields. The comparison. It conveniently uses the Raspberry-PI and
UN gauges the total populace will ascend from 7.3 billion python platforms to make the recognition task easier. It
today to 9.7 billion out of 2050. The world will require is widely used for real time applications which have a
high accuracy requirement. It has defined functions
which help in detecting an object in variable lighting

1
• Proceedings for DJ SPARK 2020 ISBN:
Automated Visual Fruit Detection for Harvesting

conditions, of different sizes and different look angles.

The convolutional neural network helps in defining
surfaces and boundaries for the correct allocation of
objects and thus their identification.
Python : This general-purpose programming language. It
supports multiple programming paradigms like object-
oriented, procedural, functional and many more. It offers
a simple syntax for the developers. The advantage of
using Python is that it has multiple libraries for specific
applications. This makes a popular choice.
The Raspberry Pi camera module : It is a 5 MP camera
designed especially for the Pi. It offers a resolution of
1080p at a framerate of 30 frames/sec. The camera
module is cheap and robust and is the best option for
image processing on the Pi.
YOLOv3 : It is a variant of a popular object detection
algorithm YOLO – ‘You Only Look Once’. YOLO is an IV. RESULTS
intelligent convolutional neural network (CNN) for live
detection of objects. This algorithm uses a single neural
network for the full image, and then allocates the image into
regions and predicts binding boxes and probabilities.

We are using YOLO because it achieves high accuracy V. CONCLUSION

while also being able to run in real-time. The algorithm
looks looks once at the image which means that it
requires only one forward propagation pass through the
neural network to predict values. After non-max
suppression (which makes sure the object detection
algorithm only detects each object once), it then outputs VI. SOCIAL IMPACT
the recognized objects together with the bounding boxes. Today there has been innovation to mechanize most of the
With YOLO, a single CNN can simultaneously predict human work however it is least in the field of Agriculture.
multiple bounding boxes and probabilities of classes for The agribusiness area in India endures most considering its
those boxes. It trains full images and optimizes the out-of-date strategies for reaping which require heaps of
detection performance directly. labour and results in helpless yields. At the core of this
The processor speed ranges from 700 MHz to 1.4 GHz wonder is the requirement for essentially expanded creation
for Pi 3 Model B+ yields. The UN gauges the total populace will ascend from
7.3 billion today to 9.7 billion of every 2050. The world will
The Raspberry Pi 3B+ features dual-band IEEE require significantly more food, and ranchers will confront
802.11b/g/n/ac WiFi, Bluetooth 4.2, and Gb Ethernet
genuine strain to stay aware of interest. Indeed, even in the
current situation at the point when the Lockdown was
forced in India vegetables like onions were fit to be
collected however because of deficiency of labour, the
yields could not be reaped at the opportune time which
brought about a cross country lack of onions and made their
costs rise fundamentally. Taking a gander at the
circumstance we were roused to fabricate an automated arm
with a productive PC vision component to specifically cull
ready organic products accordingly diminishing labour. In
this manner, building up a computer vision calculation
which can be utilized with an automated arm can expand
exactness, lessen work and time.

Block Diagram:

Method 1A:

2
• Proceedings for DJ SPARK 2020 ISBN:
Automated Visual Fruit Detection for Harvesting

Applied Laplacian transform for edge detection. Other edge

detection method were also used like Canny, SobelX,
SobelY and Scharr for edge detection. Comparing results
we saw Laplacian provided best results.
In image processing detecting the shapes is done with the
help of contours. Contours are basically all the points along
the boundary of an object in the image with the same colour
or intensity. Upon detecting the contours, the number of
vertices play an important role in identifying whether the
object of interest is a square or a rectangle or a circle. There
is an available function in OpenCV to find contours in the
image. The contours are points along the boundary and
these are stored in the memory. We can define the contours
by only the vertice points without storing all the pixels
Method 1B: along the boundary, this process is used in almost all shape
detection applications and is known as contour
approximation.
minEnclosingCircle is used to find coordinates of center and
radius of circle made on contour. drawContours() is used to
draw contours. Contour only gets displayed if the area of
contour is greater than or equal to threshold area, thus
neglecting all small unnecessary contours in the image.

ACKNOWLEDGMENT
This project and technical paper was greatly supported by
Prof. Aarti Ambedkar. Her appropriate mentoring and total
guidance helped us at every stage of this project and this
Implementation: paper would not have been possible without his valuable
This is the image processing method. We will be extracting contributions. We thank our colleagues who provided
various features from the images in the dataset so that our insight and expertise that greatly contributed to the research
model will learn to classify the fruits as per our requirement. and helped us with the entire process.
Some features are mentioned below :- The following was
carried out to detect a Tomato and Apple. We start by
converting the rgb image to gray scale. REFERENCES
Using kernel of 7 by 7, we applied median blur is used to 1. “APPLE DEFECT DETECTION USING DEEP
reduce salt and pepper noise. We tried using different LEARNING BASED OBJECT
blurring techniques and found median blur as most DETECTION FOR BETTER POST HARVEST
appropriate. HANDLING” Paolo F.
To detect the colour, a mask is applied on the captured Valdez
frame such that only the desired colour i.e. colour of fruit is 2. “Using YOLOv3 Algorithm with Pre- and Post-
highlighted. The captured image is in the RGB colour space Processing for Apple Detection in Fruit-
and is converted into the HSV colour space. This helps in Harvesting Robot” Anna Kuznetsova, Tatiana Maleva and
highlighting the single colour and blocking all other colours Vladimir
from the image. The mask will be further divided into two Soloviev.
types, one for the lower intensity levels and one for the 3. You Only Look Once: Unified, Real-Time Object
higher intensity levels. We also used trackbar as an Detection” Joseph Redmon , Santosh
alternative approach to get the desired value of red in HSV. Divvala , Ross Girshick , Ali Farhadi, University of
For improving the accuracy of contours morphological Washington , Allen Institute for AI , Facebook AI
transformation is used on the image. Applied morphological 4. Baeten, J.; Donné, K.; Boedrij, S.; Beckers, W.; Claesen,
operation closing to remove holes inside fruit image. E. Autonomous Fruit Picking Machine:
Applied morphological operation opening to remove A Robotic Apple Harvester. In Field and Service Robotics;
external image noise. Since morphological transformation Springer Verlag: Berlin, Germany,
had a certain time lag in real time we used simple 5. Bargoti, S., and Underwood, J. Deep fruit detection in
thresholding along with image dilation. dilation function is orchards. In 2017 IEEE International Conference on
used to remove noise from the image. This along with canny Robotics and Automation (ICRA) (May 2017), pp. 3626–
edge detection gave a output with negligible time lag in real 3633.
time.

3
• Proceedings for DJ SPARK 2020 ISBN:
Automated Visual Fruit Detection for Harvesting

6. Chan, T. F., and Vese, L. A. Active contours without

edges. IEEE Transactions on Image Processing 10, 2 (Feb
2001), 266–277.
7. Ciresan, D. C., Meier, U., and Schmidhuber, J. Multi-
column deep neural networks for image classification.
CoRR abs/1202.2745 (2012).
8. Kapach, K., Barnea, E., Mairon, R., Edan, Y., and Ben-
Shahar, O. Computer vision for fruit harvesting robots –
state of the art and challenges ahead. Int. J. Comput. Vision
Robot. 3, 1/2 (Apr. 2012)
9. Keras. Keras. [Online; accessed 19.05.2020].
10. www.python.org
11. www.opencv.org
12. www.wikipedia.org
13. www.paperswithcode.com
14. www.medium.com
15. www.towardsdatascience.com

4
• Proceedings for DJ SPARK 2020 ISBN: