International Conference on Communication and Signal Processing, April 3-5, 2018, India

Visual Assistance for Blind using Image

Processing
Deepthi Jain B, Shwetha M Thakur and K V Suresh


Abstract—Visually impaired people face lot of difficulties in the necessary sign boards like washroom.
their daily life. Many a times they rely on others for help. Several The paper is organized into six sections. Section I provides
technologies for assistance of visually impaired people have been a brief introduction about the proposed system. Section II
developed. Among the various technologies being utilized to assist
the blind, Computer Vision based solutions are emerging as one of
presents inferences drawn from carrying the literature survey.
the most promising options due to their affordability and The proposed system is discussed in Section III. The
accessibility. This paper proposes a system for visually impaired implementation and algorithm of the proposed system is
people. The proposed system aims to create a wearable visual aid presented in Section IV. Section V gives the snapshots of the
for visually impaired people in which speech commands are results obtained with the proposed system. The paper is
accepted from the user. Its functionality addresses identification concluded with Section VI.
of objects and sign boards. This will help the visually impaired
person to manage day-to-day activities and to navigate through
his/her surroundings. Raspberry Pi is used to implement artificial II. LITERATURE REVIEW
vision using python language on the Open CV platform. A conventional method that has been used for decades for
assistance of the visually impaired is the use of guide dogs that
Index Terms—Haar-like Features, Image Processing, Indoor
are trained to help the blind to navigate and walking cane for
Navigation, Open CV, Python, Raspberry Pi, speech commands,
Video capturing. avoiding obstacles. Smart rehabilitative shoes and spectacles
proposed in [2] facilitate safe navigation and mobility of blind
I. INTRODUCTION individuals. Each shoe is mounted with ultrasonic transducers
to detect objects at different heights. The spectacles are
LINDNESS is a problem that plagues millions of people
B everywhere. Blind people face many types of hurdles in
performing every day routine works. Even in their own
instrumented with a pair of ultrasonic transducers mounted
centrally above the bridge, and with a buzzer at one of the
temples. The cane proposed in [3] uses a ping ultrasonic
homes they must exhibit efforts to navigate from one place to
sensor along with Camshaft Position Sensor (CMP) compass
another and to locate objects. According to the World Health
sensor 511 in order to provide information about potholes and
Organization (WHO), 253 million people live with visual
hitches. The disadvantage of this system is that the CMP
impairment, 36 million of which are blind and 217 million
compass sensor is susceptible to any stray magnetic fields and
people have moderate to severe vision impairment [1]. The
it is affected by any iron objects in the surroundings. It
conventional methods adopted like cane helps in avoiding the
provides long distance navigation but is not helpful in indoor
obstacles in their way but they do not help them identify and
navigation. Earlier, sensor based approach were just to detect
locate the objects. Hence, assistance is required for the blind
objects rather than recognizing them. Thus, image processing
that helps him/her in locating objects in an indoor
proves to be a viable method in handling these situations.
environment.
A Stereo Image Processing System for Visually Impaired is a
In this context, the current work proposes a novel solution
system that includes a wearable computer, stereo cameras as
for assisting the blind. The proposed system intends to assist
vision sensor and stereo earphones, all mounted on a helmet.
the blind by taking voice commands to detect objects using The image of the scene in front of visually handicapped is
image processing and provide audio output to navigate and captured by the vision sensors. The captured images are
reach the required object. The system also intends to recognize processed to enhance the important features in the scene in
front, for navigation assistance. In order to incorporate the
distance information, stereo cameras are used [4]. But, the
Deepthi Jain B is with the Siddaganga Institute of Technology, Tumakuru,
Karnataka, India. (phone: 8970826718; fax: 2282994 ; e-mail: system uses a stereo camera hence making the system complex
[email protected]). and cost ineffective. Real-Time Visual Recognition with
Shwetha M Thakur, is with Siddaganga Institute of Technology, results converted to 3D Audio is a system which comprises of
Tumakuru, Karnataka, India. (Phone: 7411532385; e-mail: several modules. Video is captured with a portable camera
[email protected]).
K V Suresh is a Professor at Siddaganaga Institute of Technology, device (Microsoft Kinect, or GoPro) on the client side, and is
Electronics & Communication Department, Tumakuru, Karnataka, India.(e- streamed to the server for real-time image recognition with
mail: [email protected]).

978-1-5386-3521-6/18/$31.00 ©2018 IEEE

0499
existing object detection models [5]. This system requires time board. It uses a 64-bit quad-core ARMv8 architecture based
to process which does not make the system to be running/ CPU. The raspberry pi version features 1GB of RAM. It uses
functioning in real time. A Haar Cascade based object an SD card to store the OS. It has USB port through which
identification is the most widely used technique for object USB microphone is connected and speech input is given. It has
identification. This classifies the whole image into significant a CSI port through which Pi camera is connected. The
and non-significant categories using a supervised technique Raspberry PI features a 3.5 mm universal headphone jack for
[6]. Color based object identification is a technique which uses audio out. The Raspberry pi performs the task of taking video
color features in identifying and tracking the object [7]. input, converting it to frames, does suitable image processing
The existing approaches suffer from the drawbacks such as in Open CV platform using Python language.
requirement of several sensors, system not being portable and
fail to do real time processing. In the proposed system, sincere B. Pi camera
attempts are made to address few of these issues. Raspberry Pi camera module is used to take high resolution
video, as well as still images. It has a resolution of 8 megapixel
III. PROPOSED SYSTEM and 30 frames per second (fps). The output from the camera is
The block diagram of the proposed system is shown in fed to Raspberry Pi for further processing.
Fig. 1. C. Ultrasonic Sensor
An Ultrasonic sensor measures the distance to an object by
using sound waves. It does so by sending out a sound wave at
a specific frequency and listening for that wave to bounce
back. The elapsed time between the sound wave being
generated and the wave bouncing back is recorded and the
distance between the sonar sensor and the object is calculated.
In this system, it is used to notify the user about any obstacle
that is ahead of him/her.
D. Open CV and Python
It is a library of programming functions mainly aimed at real
time computer vision. It is used for various applications such
as augmented reality, gesture recognition, feature matching
etc. It is imported by using the command “import cv2” in
python. Python is a widely used high level programming
language which has a dynamic type system and automatic
memory management and supports multiple programming
paradigms including object oriented, imperative, functional
programming and procedural styles. Python is a light-weight
programming tool that has many built-in functions and does
not consume many resources while operating on the Raspberry
Fig. 1. Block diagram of proposed system pi.

The system uses speech commands as the user interface. A IV. IMPLEMENTATION AND ALGORITHM
microphone is used to capture speech input. The obtained
The video is captured using a Camera which is then divided
input is recognized using Google API. Also, it uses image
into a sequence of frames. Object detection is done using Haar
processing as its primary technique to identify objects and sign
cascade classifiers and color based object detection technique.
boards. Video is captured by the Pi camera, from which the
Haar Cascade Algorithm: The Open CV library in Python
frames are extracted. The frames are preprocessed for better
has functions specifically to detect objects. It delivers software
results. Image processing algorithms for object detection are
packages that are used to train classifiers for their object
applied on these frames and the object is detected. After the
detection system, called Haar Training.
object is located, audio messages from headsets are given to
Haar-like features: Object Detection using Haar feature-
the user to notify him/her about the location of the object.
based cascade classifiers is a machine learning based approach
Obstacle detection is done using ultrasonic sensors. Any
where a cascade function is trained from a lot of positive and
obstacle encountered in the path is notified to the blind person
negative images. It is then used to detect objects in other
by producing a beep sound from the buzzer.
images. The algorithm extracts images using a lot of positive
A. Raspberry Pi and negative images. A Haar-like feature can be considered as
The raspberry pi is a single-board computer based on a template of several white and black rectangles
Broadcom BCM2837 system on chip. It has a 1.2GHz CPU on interconnected. The features used are different size and

0500
rectangular. The main Haar-like features are shown in Fig. 2. Where, are spatial moments. The and
coordinate used in drawing the circle help us to track the ball.
The flow graph of the system for determining the position of
the object is depicted in Fig. 3. Decision is made based on the
speech command given. If the speech command is for tracking
particular object or sign board, the function for the
corresponding object or the sign board is invoked.

Fig. 2. Feature prototypes of simple Haar-like and centre surround [8].

The features value for the given mask is calculated as the

weighted sum of intensity of the pixel intensities covered by
the whole mask. But all the features extracted will not be
useful for the purpose at hand. For example, in this paper,
objects like washroom, switch board, banana are relevant. At
first, images with object of interest (positive image) and
images void of object of concern (negative image) are
captured. Images are resized and converted to gray scale using
(1).

Where, R, and are the Red, Green and Blue components in

a pixel of an image.
Text files of positive and negative images are generated
which contain the coordinates of the positive and negative
training samples along with their names in a specified order.
Text files so created are used to obtain a vector file (.vec).
This file contains compact information of positive instances of
objects and negative image such as background. The training
of classifiers is done using the vector file. After the training is
completed, the output is stored in a directory in the form of
text files. Finally, these files are converted into a single xml
file which is the final classifier. The same classifier is used in
python code to detect as well as track the object. Commands
like ‘left’, ‘right’ and ‘center’ given via headsets aid the blind
man to reach the object.
Color based object detection: Some colored objects like a
green bottle; a red ball is detected and tracked using color
based object detection technique. At first, the upper and lower
boundaries of color of the object are defined in HSV color.
Video is captured and the same in converted to frames. These
frames are converted into HSV color format.
Then a filter is created which will mask the green color. The
image obtained will be a binary image and it is eroded and Fig. 3. Flowgraph for object detection
dilated to remove noise. Then the largest contour in this binary
image is found and then the coordinates and radius of a circle V. RESULTS
enclosing this contour is obtained. A centroid is determined
The proposed system focuses on detection of objects. The
using (2) and (3).
system is made wearable and is portable. The system is
mounted on the chest of the person. The Pi camera connected
to Raspberry Pi captures the video of the scene and this is
converted into frames by the processor. The user needs to
provide speech commands when he/she is in need of the
object. The speech input is recognized and the system guides

0501
the user to the object via audio output. Fig. 4, Fig. 5 and Fig. 6
shows an object (green bottle) being detected and guiding the
user whether the object is to his right, left and center
respectively. Fig. 7 shows a banana being detected. Fig. 8
depicts washroom sign board being detected. Fig. 9 shows
snapshot of a chair detected.

Fig. 7. Snapshot of banana detected.

Fig. 4. Snapshot of bottle towards the right as seen from Pi camera.

Fig. 8. Snapshot of washroom sign board detected.

Fig. 5. Snapshot of bottle towards the left as seen from Pi camera.

Fig. 9. Snapshot of a chair detected.

VI. CONCLUSION
This paper presents a novel technique for assisting visually
impaired people. The proposed system has a simple
architecture and makes it user friendly thus, making the subject
independent in his/her home. The system also aims at helping
blind to navigate in his/her surroundings by detecting
obstacles, locate his basic necessities, read sign boards and
Fig. 6. Snapshot depicting bottle at the center as seen from pi camera. texts. Preliminary experiments show promising results as the
user can freely navigate in his surroundings safely. The system
is made much more user friendly by accepting speech as the
input to access his basic necessities.

0502
 REFERENCES [5] Rui Jiang, Qian Lin Li, “Let Blind People See: Real-Time Visual
Recognition with Results Converted to 3D Audio”, Proc. International
[1] Vision impairment and blindness: Fact sheet Available: Conference on Computer Vision, 2015.
http://www.who.int/mediacentre/factsheets/fs282/en/. [6] M. Hiromoto, H. Sugano, and R. Miyamoto,“Partially Parallel
[2] Ziad O. Abu-Faraj, Paul Ibrahim, Elie Jabbour and Anthony Ghaoui, Architecture for AdaBoost-Based Detection With Haar-Like Features”,
“Design and Development of a Prototype Rehabilitative Shoes and IEEE Trans. Circuits and Systems for Video Technology, vol. 19, Jan
Spectacles for the Blind”, IEEE Int. Conf. BioMedical Engineering and 2009, pp. 41-52.
Informatics, 2012, pp. 795-799. [7] Pritpal Singh, B.B.V.L. Deepak, Tanjot Sethi and Meta Dev Prasad
[3] Giva Andriana Mutiara, Gita Indah Hapsari and Ramanta Rijalul, Murthy ,“Real Time Object Detection and Tracking Using Color
“Smart Guide Extension for Blind Cane", IEEE Int. Conf. Information Feature and Motion”, IEEE Int. Conf. Communication and Signal
and Communication Technologies, 2016. Processing, 2015.
[4] G. Balakrishnan, G. Sainarayanan, R. Nagarajan, and Sazali Yaacob, “A [8] Rainer Lienhart and Jochen Mandt, “An Extended Set of Haar-like
Stereo Image Processing System for Visually Impaired”, Int. Journal of Features for Rapid Objection Detection”, IEEE Int. Conf. Image
Computer, Electrical, Automation, Control and Information Procesing, 2002.
Engineering, vol.2, No.8, 2008, pp. 2794-2803.

0503