Robot navigation through Qr and Text detection

* 1 2 3 4
Rajesh Kannan Megalingam , Sathi Lakshmi Madhuri , Tangudu Santhoshini , and Varsha Juluri
1,2,3,4Electronics and Communication Engineering

Amrita Vishwa Vidyapeetham

Kollam, Kerala, India
1
[email protected]
2
[email protected]
3
[email protected]
4
[email protected]

Abstract. In the rapid advancement of today’s technological world, we believe smarter technology can solve
problems. QR code detection and text recognition are one of the extensively using applications. So,
retrieving information using these applications accurately is one of the important factors. In this paper
accuracy rate of both the applications are briefly characterized in terms of time consumption for retrieving
information by varying distance. The experimental setup includes camera fixed on the robot vehicle and
navigates as per the given QR and text instructions. The accuracy rate is analysed with respect to the
response time of the robot. Finally, we have briefly discussed the performance of both the approaches.
Keywords: Accuracy, ros, open cv, python, qr code.

1 Introduction

QR code detection is playing an important role in storing information for various applications. It is essential
feature for applications such as advertisements, smart phones, banking and data transfer. There are several
applications to store information which are similar to QR codes. Among them, the bar code is also one such
application to retrieve data. The main difference between these two applications is storage capacity. The bar code
and QR code constitutes of 1-D and 2-D matrix respectively. Due to this high spatial dimensional, the QR codes
can store large amount of information compared to bar codes. Because of high storage capacity, the QR code
usage is increasing various applications. The QR codes stores information in grid structure with black and white
patterns.
At the same time, the text detection using camera’s is also placing an important role in today’s applications
such as bank cheque books, automobiles and number plate recognition. The input image is pre-processed to
remove noise and then image is partitioned into segments which makes analysis easy. And then features are
extracted from the image. These features are sent into a classifier, which classifies extracted features by
comparing with the features in its training set. In this work, we classify given data using tesseract. The tesseract is
one of the most efficient text recognition deep learning algorithm which classifies data with the help of LSTM
classifier. These two data retrieving approaches are playing an important role in robot navigation and
automobiles. So, in this paper we would like to analyse accuracy of these two approaches to navigate robot
effectively.
The experimental setup includes camera fixed on the robot vehicle and navigates as per the given QR and text
instructions. The bot is controlled using scanning QR code and text recognition. The accuracy rate is analysed
with respect to the response time of the robot. Finally, we have briefly discussed the performance of both the
approaches. The camera fixed to the navigating bot recognizes text and extracts data from QR code and navigates
as per the instruction. The robot is controlled using ROS platform.

2 Motivation

In today’s world, data retrieving approaches are playing an important role in robot navigation. There are several
applications which uses QR detection and text recognition in navigation of robot. Both text recognition and data
extraction from QR have their individual benefits. In this paper detail analysis is carried out on the robot response
time using QR and text recognizing. This response time also helps for pre-analysis of various applications like
time consumption of data entry tasks, billing, passport verification and banking. The text detection using
camera’s is also placing an important role in today’s applications such as bank cheque books, automobiles and
number plate recognition. Due to less time consumption for data retrieving from Qr code we designed a system
by utilising Qr codes. In this modern era the qr codes usage is increasing rapidly on daily basis . Users can
generate or create their own QR code based on the requirement, which can be accessed by the smart phones. By
scanning the qr code we can decrypt the data or link interconnected with it, then that related link will directly
bridge to the web and obtain the corresponding information. Mobile technologies have great ability to create
Quick Response (QR) codes and also have appreciable impact on decoding the information with in qr codes.

3 Related works

In research work [1], the author briefly discussed the introduction on QR, explained the various aspects of QR
code applications and 2D matrix design of QR code which have capacity to store large amount of data and
decoded at high speed. The authors in the research article [2] focused on QR code application in urban traffic
control and concentrates on various novel strategies to implement a system to upgrade the supervision of car
parking in urban areas. The authors in paper [3] describes a cost-efficient application of QR code in hospitals for
the identification of patients and for storing their medical information and, they developed the system to access
the medical data network in turkey using QR identity tag. The authors discussed the usage of QR code in different
research fields and highlight some of the secured aspects of QR code control the various applications and dealt by
QR code in real time approach for verification of vehicles and its identification in paper [4].
The author [5] introduces a methodology for generating QR code where the user enters text in a web browser
and obtains the QR code on single and multiple lines of both English and Thai languages. The implementation of
computer-based applications for the easy readability of text from images by examining the different methods of
text recognition is proposed in [6]. In a research article [7], the author focused on detecting texts regions using the
method of scene text recognition and the development of application-based text extraction in smartphones. In [8],
the author gave a brief explanation on text recognition and some standard datasets, in addition to it he reviews
retrieving information from the image through various machine learning algorithms to detect and differentiate the
methods. The work published in [9], delivers the survey on the predictive big data analysis and accuracy it
provides in the health care system and also detailed the tools that are engaged in making possible the forecast
and the also reports the challenges and the applications of it.
In paper [10],the author explained the methodology used for the classification of the text involving the
process of identification and categorization using the multi task learning and applies the gated multi task to filter
the useless and further uses the Caps Net with the routing based on task to elude the interferences in the tasks .
The author in [11] aims to introduce, Simulation Localization and Mapping (SLAM) and to construct a 2D
mapping robot by integrating a robot operating system and Arduino, in order to develop own robot for 2D
mapping. The authors in [12], discussed the challenges of robot software systems of interacting each other in
systematic way of communication between nodes and their capabilities uses in standard robot applications. In the
article [13], the authors presented an efficient way of nodes communicating with each other during the simulation
of various poses for a humanoid robot using the ROS platform.
In [14], the author focused on development, modification and on the scalability of the Robot operating system
which allows various nodes to work simultaneously. The authors propose the management on wastage in
effective manner to reduce the pollution and global warming on IOT based application, which executed using
ROS (Robot Operating System) in paper [15]. In this paper, we developed a systematic design to compute the
time required for retrieving data from the QR code and text. And have chosen well-defined algorithms for text
detection and QR detection. And the acknowledgment of robot as per the commands is noted.

4 System architecture

The system architecture of QR and text detection-based control system using ROS is depicted in Fig. 1. In this
research work accuracy rate of both the applications is briefly characterized in terms of time consumption for
retrieving information by varying distance. The experimental setup includes a camera fixed on the robot vehicle
and navigates as per the given QR and text instructions using ROS. There are mainly three modules in both the
architectures. The three modules are briefly explained below.

4.1 Camera Capture

The input to the camera can be a real-time video or an image. The real-time capture of frames from the video will
be processed further by pyzbar or tesseract. In this work, we used a Logitech webcam to capture an image.

Fig. 1. Architecture of proposed design.

4.2 Pyzbar

The pyzbar is a widely used application to process the QR code and bar code. We used pyzbar to decode Qr code.
In our research work the qr codes which are encoded with instructions are used for robot navigation which is
shown in detail in Fig.1. The process involves binary conversion, location, segmentation and decode. Our aim is
to locate qr in an image which is acquired as depicted in Fig. 2. Binary conversion is used to convert the
information of qr code into binary form. These binary forms can easily understand by system and makes the
further process without delay. From the first two blocks the wanted section is obtained by mapping the
corresponding section pixels with one. The located QR code is segmented. The located data is segmented in to
blocks of equal size. Segmentation makes the analysis easy where each block can easily decode without
interruptions of each other. The time taken to retrieval data using pyzbar is too fast because when we scan qr,
without any delay we can see information is displayed on screen and as there is no loss in data the accuracy rate
of pyzbar is 100%.

Fig. 2. Step wise implementation of pyzbar.

4.3 Tesseract

The input image is pre-processed to remove noise and then partitioned into segments which makes analysis easy.
The features are extracted from the image and sent to classifier to extract features to predict data. It recognizes the
text in the image with accuracy rate of 50-70%.

4.4 Robot Navigator

The message after decoding the qr code or recognising text is received by the Arduino and checks with the string
given in the code. If received message returns zero with the compared string then the corresponding case will be
executed. The robot navigates as per the command given in the executed case. And if not returns zero with any of
the string then robot stops as its default case. The algorithm of robot navigator is clearly depicted in below
flowchart Fig. 3.

Fig. 3. Flowchart for Detection of QR and Text.

5 Design and implementation

The hardware setup depicted in Fig. 4 consists of Arduino, motor driver, chassis and dc motor. The robot vehicle
navigates as per the given QR and text instructions.

Fig. 4. Wheeled bot.

 Fig.5. Detection of QR through cam.

The implemented software detects the QR code or Bar code in the captured photo. This is done using ros-
OpenCV where libraries need to be installed to identify QR codes or Bar codes in the captured frames. With the
help of pyzbar module QR codes or Bar, codes are identified and perfectly decoded. In this project we created
some of the QR codes which are encoded with left, right, forward, backward, finish. When these generated QR
codes are captured by the Logitech camera, it is decoded, and data is transmitted through ros topic. This Ros topic
is programmed in a way that it carries a decoded message to the controller. The rate of the message can be
adjusted as per the need. The rate can be controlled by programming ros topic. The decoded QR code or text
image in the ros topic is printed on the terminal with webcam is shown in Fig. 5. When the cam detects QR code
containing “backward” from a particular distance it displays the data “backward” in the terminal. The output
terminal is shown in Fig. 6.

Fig. 6. Output terminal while detecting QR.

Fig. 7. Recognition of Text through cam.

 The implemented software also detect text in the captured photo. This is done by using ros-OpenCV which
supports an optimized algorithm to identify text in the captured frames. In this work with the help of tesseract v4,
the text in the captured frame is detected and recognized. When the printed text such as left, right, forward,
backward and stop captured by the camera and then published through a ros topic to controller subscriber. The
“right” text in the input image is recognized by the camera as shown in Fig. 7. And the output terminal while
recognizing text is shown in Fig. 8. A brief description of various hardware and software modules are explained
below.

Fig. 8. Output terminal while detecting text.

5.1 Robot operating system (ROS)

Generally, ROS is used for the software development of robots. It is used for the start of communication among
multiple nodes.” Node” is the ROS term which is connected to the ROS network through ROS master. Here we
have publisher node which continually broadcast messages to the subscriber node through ROS topic. In our case,
message from camera reaches the other node i.e. to Arduino. Even though ros topic and nodes are registered, they
will be able to communicate with each other only with the help of roscore (Ros master).

5.2 OpenCV (open source computer vision)

OpenCV was setup to furnish a normal foundation for computer vision solicitation and to speed up the use of
machines. It is a library consist of 2000 to 3000 optimized and advanced algorithms. These algorithms are used
for many applications like to detect or to recognize objects or humans as well as to extract the data from the given
image. The algorithm used for text recognition is tesseract OCR engine. This algorithm is able to precisely
decrypt and extract text from a different form of sources. so tesseract has a straightforward programme in
decoding images. Tesseract is capable of extracting of different texts and also supports over 100 other languages
as well. This algorithm works best when the image only contains text and is on a single line.

5.2.1 Qr code recognition

Pyzbar is a python library, which identifies if any qr codes or bar codes present in the captured image and then
decodes the information present in QR code or bar code.

5.2.2 Text recognition

Tesseract is an OCR engine with support for Unicode. The captured image is trained by tesseract, later the text
in image can easily converted to string. Here we used latest version of tesseract i.e., v4. It has the ability to
recognize more than 100 languages. We can also create training set to improve its application. This includes pre-
processing technique which removes noise. The technique is binarization which converts an image into a black
and white image. Then it is segmented to make analysis easy. The lstm classifier classifies each character by
comparing with its existing training set.
5.3 Arduino UNO

It is an open-source microcontroller, which acts as an interface between electronic devices. The board consists
of 14 digital and 6 analog I/O pins (input/output) and programmable with Arduino IDE (Integrated Development
Environment) through a type B USB cable. It provides 3.3v and 5v for electronic appliances. limited I/p voltage
is 6v-20v and recommended voltage is 7v-12v.

5.4 Wheel bot

It is built with dc motors, which are powered by the motor driver. Motor driver is used to control the speed and
rotation of motors which also acts as an interface between Arduino and motors.

6 Experiment and results

The experiment is done using a webcam of 720 pixels. This is carried out for various distances from the camera
and carried at normal light conditions. Our experimental setup included the QR code encoded with “backward”
and the “right” printed as text on a white paper. The accuracy rate is analysed with respect to the response time of
the robot. The experimental results are based on the analysis of time taken to decode QR code encoded with
“backward” recognise “right” printed as text on a white paper and to initialise robot navigation. It is noticed that
the time for the extraction of data from the QR code is almost constant for every distance with minute variation as
shown explicitly in Fig. 9. Time for text recognizing becomes unstable by varying distances which is shown
clearly in Fig.10. We observed that time taken for robot initiation is more in high light conditions when compared
to normal light conditions in both the cases of text and Qr codes. Qr codes are decoded even if orientation of qr
code is changed anywhere below its threshold and Text is not recognised if orientation of it is changed. From fig
9 and 10, we can conclude that time taken for data retrieval from Qr codes is almost constant at any distance
below its threshold and in case of text recognization time changes at every distance below its threshold. From the
experiment result, we can conclude that the time taken for the data retrieval from the QR code using pyzbar is
lesser and more stable than the text using tesseract. The image resolution is low when the object is beyond its
threshold.

Fig.9. Experiment results of QR Detection.

Fig. 10. Experiment results of Text Recognition.

7 Future works

From this project we can say that qr detection using pyzbar is faster method than text recognition using tesseract.
With low cost the present bot can be modified to automatic wheelchair with some micro controllers and some
sophisticated sensors to avoid obstacles in their path. It can be automated as wheelchair for paralyzed persons to
reach their destination. This automated wheelchair can be equipped with camera to scan the qr code which helps
them to reach the destination. And secured qr codes can be implemented by using websites that identify real
users. The qr code with important information can be scanned by only those users have authentication details to
access it, by which we can increase the security for these qr codes.

8 Conclusion

We successfully differentiated the accuracy(speed) of data extraction and subscription by using ROS, open cv and
Arduino IDE. Based on the experimental results it has been found that out of the two methods qr and text, the
speed of extraction of the content present in qr code is faster than text recognition in the given image or video.

Acknowledgement
We would like to express our gratitude towards God and our mentor Dr. Rajesh Kannan Megalingam sir Director
of Hut Labs for this great opportunity. We thank each faculty of ECE department for their support and giving us
good acknowledgement on basics of electronics.

References

• S. Tiwari, "An Introduction to QR Code Technology," 2016 International Conference on Information Technology (ICIT),
Bhubaneswar, 2016, pp. 39-44.
• A. Bechini, F. Marcelloni and A. Segatori, "A mobile application leveraging QR-codes to support efficient urban
parking," 2013 Sustainable Internet and ICT for Sustainability (SustainIT), Palermo, 2013, pp. 1-3.
• V. Uzun, "QR-Code Based Hospital Systems for Healthcare in Turkey," 2016 IEEE 40th Annual Computer Software and
Applications Conference (COMPSAC), Atlanta, GA, 2016, pp. 71-76.
• K. Saranya, R. S. Reminaa and S. Subhitsha, "Modern applications of QR-Code for security," 2016 IEEE International
Conference on Engineering and Technology (ICETECH), Coimbatore, 2016, pp. 173-177.
• Sutheebanjard, P. & Premchaiswadi, Wichian. (2010). QR-code generator. 89-92. 10.1109/ICTKE.2010.5692920.
• Manwatkar, Pratik & Singh, K.. (2015). A Technical Review on Text Recognition from Images.
10.1109/ISCO.2015.7282362.
• C. Yi and Y. Tian, "Scene Text Recognition in Mobile Applications by Character Descriptor and Structure Configuration,"
in IEEE Transactions on Image Processing, vol. 23, no. 7, pp. 2972-2982, July 2014.
• Y. Sun, A. Dawut and A. Hamdulla, "A Review: Text Detection in Natural Scene Image," 2018 3rd International
Conference on Smart City and Systems Engineering (ICSCSE), Xiamen, China, 2018, pp. 826-829.
• Smys, Smys. (2019), “SURVEY ON ACCURACY OF PREDICTIVE BIG DATA ANALYTICS IN HEALTHCARE,” Journal of
Information Technology and Digital World. 01. 77-86. 10.36548/ijtdw.2019.2.003.
• Jacob, Dr. (2020),” Performance Evaluation of Caps-Net Based Multitask Learning Architecture for Text Classification,”
Journal of Artificial Intelligence and Capsule Networks. 2. 1-10. 10.36548/jaicn.2020.1.001.
• A. L. Ibáñez, R. Qiu and D. Li, "A Simple, Cost-Effective and Practical Implementation of SLAM Using ROS and Arduino,"
2017 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications
(GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), Exeter, 2017,
pp. 835-840.
• W. Woodall, M. Liebhardt, D. Stonier and J. Binney, "ROS Topics: Capabilities [ROS Topics]," in IEEE Robotics &
Automation Magazine, vol. 21, no. 4, pp. 14-15, Dec. 2014.
• H. Feng, C. Wong, C. Liu and S. Xiao, "ROS-Based Humanoid Robot Pose Control System Design," 2018 IEEE
International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan, 2018, pp. 4089-4093.
• V. A. Hax, S. S. d. C. Botelho and N. L. D. Filho, "ROS MMaster: Adapting ROS for Better Scalability," 2014 Symposium
on Automation and Computation for Naval, Offshore and Subsea (NAVCOMP), Rio Grande, 2014, pp. 14-17.
• Megalingam, Rajesh Kannan & Sarveswara Reddy, Sathi & Pula, Bhanu & Chandrika, Dega & Reddy, Nara. (2019).
Cloud Computation Based Urban Management System Using Ros. 764-769. 10.1109/ICOEI.2019.8862790.