2020 IEEE International Conference for Innovation in Technology (INOCON)

Bengaluru, India. Nov 6-8, 2020

Sorting of Objects Using Image Processing

1st Prashast Yadav 2nd Meenakshi Uikey 3rd Prajakta Lonkar
Electronics And Telecommunications Electronics And Telecommunications Electronics And Telecommunications
Vidyalankar Institute of Technology Vidyalankar Institute of Technology Vidyalankar Institute of Technology
Mumbai, India Mumbai, India Mumbai, India
prashast.yadav@vit.edu.in meenakshi.uikey@vit.edu.in prajakta.lonkar@vit.edu.in

4th Sachin Kayande 5th Amit Maurya

Electronics And Telecommunications Electronics And Telecommunications
Vidyalankar Institute of Technology Vidyalankar Institute of Technology
Mumbai, India Mumbai, India
sachin.kayande@vit.edu.in amit.maurya@vit.edu.in

Abstract—Sorting is a regular process for many industries B. Literature Survey

and small businesses. The paper proposes a low-cost automated
system using Deep Learning, Raspberry Pi 3 module B, and a For several years considerable effort has been taken to
conveyor belt system for sorting of various day to day products. improve the sorting process in various industries. Over the
Object sorting is done based on their features. The physical
implementation includes a conveyor belt and a pneumatic piston, years, this sorting was done with the help of human strug-
which is controlled by the Raspberry Pi for the sorting of the gles that brought up many disadvantages like increased time
object. TensorFlow, with some libraries and methods, is used for consumption, and human errors. To overcome these problems,
the detection of the object. many researchers have proposed various methods for the
Index Terms—Conveyor belt system, Pneumatic Cylinder, sorting process, which includes object detection and sorting
Raspberry Pi 3, TensorFlow, SSD Mobilenet, USB Camera
on multiple bases, like sorting based on color or shape or
size. In one of the previous papers, the sorting of objects was
I. I NTRODUCTION
done based on colors. Euclidean distance-based approach was
Automation improves the overall productivity in any indus- used to find the distance between average color in a region to
try. The paper suggests a new approach to develop a system that of the desired color. The major drawback of this approach
that can segregate the products placed on the conveyor belt is when the colors are mixtures between two colors, making
based on their features by building mechanical structures to it difficult for the system to recognize the object [1,2]. The
sort the same. proposed system in this paper uses a different approach to
This automated system does not require any special human overcome such issues that arise when sorting is done only
vigilance and thus reduces the probability of human errors. based on colors or shapes.
The proposed Automated Object Sorting System is devel- The basic idea for the construction of the conveyor belt and
oped using Single Shot Detector Model (SSD) with Mobilenet the sorting system was obtained from the IEEE paper entitled
feature extractor. A Deep Learning based image classification “Object Detection and Robotic Sorting System in Complex
model is used to classify objects as per requirement, based Industrial Environment”. [3]This system has designed and
on the object recognition accuracy in real-time. Deep learning developed a sorting machine with the help of deep learning.[4]
classifies objects into different categories. It used multiple lay- In some previous papers, it was documented that object
ers of data for feature extraction and representation, and once detection can be done using the SIFT approach[7]. However,
the system is trained with initial datasets, the identification of there are some drawbacks in using the SIFT approach as it
objects can be achieved with almost accuracy. uses intensive computational power due to which it cannot be
used for real-time application. Also, the SIFT approach cannot
A. Problem Statement detect objects that do not show regions of texture. [5]-[7].
Sorting of products has been reported to be a complex Real-time object detection is performed using TensorFlow’s
problem in various small scale industries and businesses. Object Detection API and OpenCV. According to the research
These businesses rely on the old mechanism for sorting these mentioned in ‘Detection of Real-Time Objects Using Tensor-
products, to reduce costs which can be utilized in various Flow and OpenCV’ it is concluded that the image processing
other productive activities. Thus, there is a need to design an done using Tensorflow is more efficient and reliable than the
automated sorting system that separates various products and, Pytorch deep learning library [8]. The paper uses the YOLO
at the same time, move objects automatically to the basket. object detection system. SSD (single-shot detection) Mobilenet
The system must be feasible, reliable, and affordable. is a convolutional neural network that has the best detection

978-1-7281-9744-9/20/$31.00 ©2020 IEEE 1

Authorized licensed use limited to: Amrita School of Engineering. Downloaded on April 23,2024 at 10:58:06 UTC from IEEE Xplore. Restrictions apply.
rate of 87.7 percent with a training to test data ratio of 80:20. D. Client-Server Communication
It has to do about nine times less work than other neural There are two communication protocols in the client-server
networks, providing the same accuracy. Thus, the suggested system; the UDP protocol and the TCP protocol [13]. In
system in this paper papers uses SSD MobileNet. UDP (User Datagram Protocol), each time the client sends
a datagram to the server, it also has to send the local socket
II. M ETHODOLOGY descriptor and the receiving socket’s address. In TCP (Transfer
A. TensorFlow Control Protocol), a connection is established between the two
sockets of client and server. The client’s socket will request
Tensorflow is the open-source library that is being used a connection while the other socket, i.e., the server’s socket,
in the proposed system for object detection. The tensors are will listen to the connection request and respond to it. Once
multidimensional arrays that consist of elements of the same the connection is established, the transmission of data can be
data type. These are a standard way of representing data in done in both the direction.
machine learning. For object detection, we provide some input In the proposed system, the TCP communication protocol
data which consists of a set of images. Using TensorFlow, is used to be more efficient than UDP[16]. The PC, where the
the model is trained using any machine learning algorithm. object detection is done, acts as a server, and Raspberry Pi 3,
The main objective of the model is to extract the main visual acts as a client. To establish a connection between the devices,
features based on edge detection and visual recognition. For they are connected to the same local network through Router.
testing the model, test data is provided, which is again a set of
images, and the final output is obtained. The output image is
then converted to Numpy array so that computation becomes
easy. TF records are also being used, which consists of images
along with the tags. TF records stores the data as a sequence
of binary strings.[16]

B. COCO Dataset
The suggested system uses a pre-trained object detection
model ’ssd mobilenet v1 coco’ on the COCO dataset, which
refers to ”Common Objects in Context.” [9],[12]. It is a
dataset used for large scale object detection, segmentation,
and captioning. This package provides Matlab, Python, and Fig. 1. Client Server Communication
Lua APIs. The COCO dataset has 330000 images, out of
which more than 200000 are labeled. It has 1.5 million object
instances, 80 object categories, 91 stuff categories, and five III. S YSTEM D ESIGN
captions per image. It also has 250000 people with key points. A. Block Diagram
[17,18]

C. SSD MobileNet
SSD is one of the popular methods used for the detection of
objects in an image. It provides scores for each object category
during screening and makes changes and adjustments to the
box if needed to match the object’s shape. This method is
easy to train as SSD has eliminated proposal generation and
feature resampling and encapsulation of all computation in a
single network. SSD is a faster method compared to other
single-stage methods.[10] Previous research has demonstrated
that SSD with Mobilenet features extractor is one of the most
accurate models out of all fastest models.[11] A significant
advantage of using MobileNet feature extractor is that it can Fig. 2. Block Diagram.
also be used in Low Powered Devices without GPU. The
efficiency of object detection using the MobileNet model is 1) Image Capturing and Processing Block:
well explained in the paper ”Real Time Object Detection Can a) USB Webcam: A USB camera captures a real-time
be Embedded on Low Powered Devices”[12]. The MobileNet image. We have used Logitech camera for this purpose. It
model makes the Object Detection process more efficient, has 720p HD video- capturing capacity, a lens of 5MP and
as well explained in the paper ”Object Sorting using Image automatic light correction features that rectifies the brightness-
Processing.[5]” related errors to give an improved viewing experience [14].

2
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on April 23,2024 at 10:58:06 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3. USB Webcam

b) PC: PC is the processing unit that takes input from

Fig. 5. Relay Module
Webcam and sends the processed results to the Raspberry Pi.
The real-time images from the Webcam are sent to the PC
3) Mechanical Block:
using the OpenCV library and processed in this unit using
TensorFlow Object Detection API. It then recognizes and
sends the data to Raspberry Pi.
2) Decision and Command Block: This block consists of
a Raspberry Pi and a relay module. Raspberry Pi acts as a
client, and it receives the data from the PC. It analyzes the
data and commands the relay module accordingly.
a) Raspberry Pi 3: “Fig 4.” shows the Raspberry Pi 3 B
module. It has 40 GPIO pins that are used to connect the relay.
HDMI cable used to connect the monitor and 5v micro USB
port it uses to give the supply to Raspberry Pi 3 B module. Fig. 6. Top View AutoCAD Model
Ethernet port is used to provide the network to the module.
The four built-in USB ports can provide output up to 1.2A,
which will enable us to connect more USB devices [15],[20].

Fig. 7. Front View AutoCAD Model

Fig. 4. Raspberrypi 3

b) Relay Module: The solenoid used to control the piston

Fig. 8. Final Conveyor Belt Setup
requires a power supply of 12V. Since the output of GPIO
pins of Raspberry Pi is 3.3V, we cannot control the solenoid a) Conveyor Belt Frame:
directly through GPIO pins. Thus, a relay module comes into
• Main conveyor frame: The material used for making the
the picture. We connect the 12V external power supply and
outer frame of the conveyor belt was 304 Stainless Steel.
the solenoid across the open circuit of the relay module and
The following two cuttings were done for the same- Two
then control the relay’s switching using the GPIO pins. The
plates of size 860x65mm One plate of size 740x200mm
relay module provides an inbuilt driving circuit that amplifies
the voltage to 5V and isolates the two circuits [14].

3
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on April 23,2024 at 10:58:06 UTC from IEEE Xplore. Restrictions apply.
 Fig. 9. Conveyor Belt Frame Fig. 11. 5/2 Solenoid Valve

d) Pneumatic Cylinder: A Double-acting pneumatic

cylinder is used in the system to sort the objects in two
• Bearing Blocks: Bearing blocks of iron were used to hold directions. A rectangular extension is attached to its piston,
the rods on both ends of conveyor belt. Four bearing which ensures that force applies to a larger portion of the
blocks of size 50x50x20mm were made. object.
• Rods and Rollers: Two iron rods of diameter 20mm were
used on both ends of the frame to support the rollers and
belt. Nylon rollers were further fitted on these rods.
• Conveyor Stands: Hollow aluminium square pipes of
dimension 130x20x10mm were used.

b) DC geared motor: DC geared motor was used to meet

the high torque requirement of the conveyor belt system. Gears
provide a balance between torque and speed.[14] Motor with
the following specifications was used: Torque: 120kgcm Input
Voltage: 12V-16V RPM: 40rpm

Fig. 12. Pneumatic Cylinder

B. System Workflow
The system has a conveyor belt that rotates at a constant
speed with 12V dc geared motor. The objects are fed at the
feed side of the conveyor belt, where the camera captures the
object’s real-time image. The image captured is then identified
and recognized by the PC using TensorFlow Object Detection
API. Then a Client-Server network is established between PC
and Raspberry Pi to send the object label as ’string’ data type.
Raspberry Pi then compares this data with the object labels
that we stored in the program. If the detected label is the
Fig. 10. DC Geared Motor
same as the stored label, then Raspberry pi makes GPIO pin
HIGH or LOW accordingly. The pins of the Raspberry Pi are
connected to the relay module, which controls the solenoid.
c) Pneumatic solenoid valve: The pneumatic solenoid The Raspberry Pi controls the switching of the solenoid via the
valve converts electricals signals into pneumatic functions. relay module. The HIGH input to the relay module will turn on
Air is directed into the solenoid as soon as the electricity is the solenoid. A pressure between 0.15MPa-0.8MPa is applied
applied. The solenoid valve used in the proposed system is a to the Pneumatic cylinder when the solenoid is switched on.
5/2-way directional valve that has two output ports, one inlet The piston then pushes the object to its respective baskets.
port, and two exhaust ports. Solenoid valve with the following
specifications is used: Model: J4V110-06, Working Pressure:
0. 15-0.8MPa.

4
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on April 23,2024 at 10:58:06 UTC from IEEE Xplore. Restrictions apply.
 Fig. 15. Bottle detection in sufficient light

• CASE-II: Under moderate lighting conditions-

Fig. 16. Cup detection in moderate light

Fig. 13. System Workflow

IV. R ESULT AND A NALYSIS

The pre-trained model used in this application shows an
accurate result of object identification. Two objects, a cup,
and a bottle, are used to test the system. The testing was done
under three different conditions of light intensity, which are
as follows:
• CASE-I: When a sufficient amount of light was available-

Fig. 17. Bottle detection in moderate light

• CASE-III: Under poor lighting conditions-

The results show that the object was detected in all the above
cases, provided the accuracy score reduced ad the amount of
light decreased. The time taken for the object to reach the
Pneumatic cylinder from the feed position is found to be 1
second. Thus a 1-second delay is taken before commanding
the GPIO pins.

Fig. 14. Cup detection in sufficient light

5
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on April 23,2024 at 10:58:06 UTC from IEEE Xplore. Restrictions apply.
 [6] S. Sumahasan, Udaya Kumar Addanki, Navya Irlapati, Amulya Jonnala,
“Object Detection using Deep Learning Algorithm CNN”, International
Journal for Research in Applied Science and Engineering Technology
(IJRASET), Volume 8 Issue VII July 2020
[7] Raghu Raj Prasanna Kumar, Suresh Muknahallipatna, John McInroy,
“An Approach to Parallelization of SIFT Algorithm on GPUs for Real-
Time Applications”, Journal of Computer and Communications(JCC),
2016
[8] Ajay Talele, Aseem Patil, Bhushan Barse, “Detection of Real Time
Objects Using TensorFlow and OpenCV”, Asian Journal of Convergence
in Technology, Volume-V, Issue-I
[9] R.S.Sandhya Devi, Vijaykumar VR , M.Muthumeena, “Waste Seg-
regation using Deep Learning Algorithm”, International Journal of
Innovative Technology and Exploring Engineering (IJITEE), Volume-
8 Issue-2S December, 2018
Fig. 18. Cup detection in poor light
[10] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott
Reed, Cheng-Yang Fu, Alexander C. Berg, “SSD: Single Shot MultiBox
Detector”, UNC Chapel Hill Zoox Inc. Google Inc. University of
Michigan, Ann-Arbor
[11] Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I,
Wojna Z, Song Y, Guadarrama S, Murphy K, ”Speed/accuracy trade-offs
for modern convolutional object detectors”, CVPR 2017
[12] Jimut Bahan Pal, Shalabh Agarwal, “Real Time Object Detection Can be
Embedded on Low Powered Devices”, International Journal of Computer
Sciences and Engineering (JCSE), Vol.-7, Issue-3, March 2019
[13] Ming Xue, Changjun Zhu, “The Socket Programming and Software
Design for Communication Based on Client/Server”, 2009 Pacific-Asia
Conference on Circuits,Communications and System.
[14] Ranjitha S Rai, Madan G, K.R Prakash, G. Ravi Krishna, “Automation
of Object Segregation”, International Research Journal of Engineering
and Technology (IRJET), Volume: 06, Issue: 06 June 2019
[15] Sushrut Nagesh Kulkarni, Sanjay Kumar Singh, “Object Sorting Au-
Fig. 19. Bottle detection in poor light tomated System using Raspberry Pi”, Proceedings of the International
Conference on Communication and Electronics Systems (ICCES 2018)
[16] Ming Xue, Changjun Zhu, “The Socket Programming and Software
Design for Communication Based on Client/Server”, 2009 Pacific-Asia
V. C ONCLUSION Conference on Circuits,Communications and System.
[17] https://blog.tensorflow.org/2020/07/tensorflow-2-meets-object-detection-
Real-time Object detection is successfully implemented us- api.html
ing a CPU with an i5 6th generation processor. The Raspberry [18] https://cocodataset.org/home
Pi controls the pneumatic solenoid and piston for sorting [19] https://github.com/cocodataset/cocoapi
[20] https://www.raspberrypi.org/
purposes. The accuracy of the recognition of objects depends
on various factors like the lighting conditions and orientation.
After object recognition, it is ensured that the pneumatic
pistons work synchronously with proper delay for the sorting
of objects. For the improvement in accuracy, the environment
consisting of constant light and monochromatic background is
used. Efficiency can also be enhanced by training our model
for a vast dataset.

R EFERENCES
[1] Sheela. S, Shivaram. K. R, Meghashree. S, Monica. L, Prathima. A,
Shriya.M.Kumar, “Low Cost Automation for Sorting of Objects on
Conveyor Belt”, International Journal of Innovative Research in Science,
Engineering and Technology (IJERSET), Vol. 5, Special Issue 10, May
2016
[2] Vishnu R. Kale, V. A. Kulkarni, “Object Sorting System Using Robotic
Arm”, International Journal of Advanced Research in Electrical, Elec-
tronics and Instrumentation Engineering (IJAREEIE), Vol. 2, Issue 7,
July 2013
[3] Liang BinYan, Wang YanBo, Chen ZhiHong, LiuJiaYu, Lin JunQin,
“Object Detection and Robotic Sorting System in Complex Industrial
Environment”, 2017 Chinese Automation Congress (CAC), Jinan, 2017
[4] Navinkumar Patle, Shweta Dhake, Devayani Ausekar, “Automatic Ob-
ject Sorting using Deep Learning”, International Research Journal of
Engineering and Technology (IRJET), Volume: 05, Issue: 08 Aug 2018
[5] Rahul Vijay Soans, Pradyumna G.R., Yohei Fukumizu, “Object Sorting
using Image Processing”, 2018 3rd IEEE International Conference on
Recent Trends in Electronics, Information and Communication Technol-
ogy (RTEICT-2018), MAY 18th and 19th 2018

6
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on April 23,2024 at 10:58:06 UTC from IEEE Xplore. Restrictions apply.