Abstract

Date:

Project Title: Object Detection and Recognition

Project Abstract:
This system explores the synergy between OpenCV's computer vision capabilities and MobileNet SSD,
a lightweight deep learning model, for efficient object detection and recognition in images. Additionally,
the integration of a Text-to-Speech (TTS) engine enriches the system by providing speech output
corresponding to the detected objects. The combined approach promises enhanced accessibility and
usability in applications such as assistive technologies, augmented reality, and smart surveillance
systems.

Project Members
Malode Om Kondaji
Malode Tejaswi Raosaheb
Neve Sujal Rajesh
Aher Mayur Anil

1
 1. Introduction

The World Health Organization (WHO) estimates that 253 million people live with visual impairment.
217 million of those have moderate to severe vision impairment and 37 million are blind. Traditionally,
Tools like stick were used from decades to avoid obstacles. Keeping pet dogs or guide dogs were one of
few techniques used by the visually impaired. Some took help of their family and friends for assistance.
However, these techniques and tools had drawbacks. People with trained dogs needed money and time
to feed and train the dogs. In today’s technologically advanced world the above techniques cannot suffice
the needs of the visually impaired people. According to statistics from the World Health Organization
(WHO), at least 285 million people are visually impaired or blindness. Blind people generally have to
rely on white canes, guide dogs, screen-reading software, magnifiers, and glasses for navigation and
surrounding object detection. Therefore, to help blind people, the visual world has to be transformed into
the audio world with the potential to inform them about objects. In this paper, we propose a real-time
object detection system to help visually impaired people in their daily life. This system helps to detect
and recognize the type of object. We propose a camera-based assistive speech output of object name to
help blind persons detect object. The Python based object detection and reader comes as a proposed
solution to enable visually impaired people to find difficulties in detecting obstacles and dangers in front
of them during walking and to identify the world around. Blind people have high difficulty in doing their
daily routines. Due to low vision or blindness, they suffer from an inferiority complex and also it affects
their economic conditions because of less efficiency in doing the work and the cost of the treatment. The
major difficulty faced by blind people while navigating or traveling in the unknown surrounding. The
ingenious device for blinds is a contraption which helps the blinds people to navigate with speed and
confidence by detecting the nearby objects and obstacles using the help of pi camera, ultrasonic sensor
and notify them with Speech warning sound along with voice alert. A portative user-friendly device is
flourished that can identify the obstacles in the path using ultrasonic sensors. If the obstacle is close then
raspberry pi sends a signal to sound a and also sends the voice command through the earphones. The
proposed system detects the obstacle images which are present in outdoor and indoor with the help of a
camera. when any objects or obstacles come in range of an ultrasonic sensor and it make notify Speech
warning messages activated when any obstacle is detected via earphone. Since the running of daily life
of blind people is very difficult. This project helps them to run their life as usual. They can make this
project as a gadget or a device in their hands which detects the obstacle. This project is more efficient
than the existing system with cheaper and accurate one.

2
 2. Objectives

The objectives are as follows:

1) It helps the sight impaired to avoid obstacles while they walk, thus avoiding accidents.
2) The main objective of this is to provide an application for blind people to detect the obstacles in
various directions
3) The main objective is that, our proposed device with advanced embedded technology which will
give the blind person an imaginary vision rather than being dependent on others. This project is
designed keeping the view of visually impaired people, with this tool they can travel to their
destination without the need of others.
4) The device will help them to go out independently.
5) Our main purpose of this product is to help blind people see, not literally but make life a little
bit easier for them.
6) Convert the information to voice using headphone
7) Connect the web camera with the system.
8) Programming the system using Python language, its powerful for processing.
9) Process and analyze the camera records using system in real time.
10) Detect and recognize objects in front of the blind.
11) Design and build an alarm system to notify the user about the recognized objects using
voice messages.

3
 3. Motivation

To enhance their independence and quality of life. By providing them with the ability to detect and
recognize objects in their environment, they can navigate their surroundings more effectively, perform
daily tasks with greater ease, and participate more fully in society. This technology can empower blind
individuals to identify objects such as obstacles, signs, tools, and even people, enabling them to make
informed decisions and interact with their environment more confidently. Ultimately, object detection
and recognition for blind people can promote inclusivity, autonomy, and accessibility in various
aspects of their lives.
Percentage of persons with disabilities in Palestinian society. Especially those with visual disabilities
(blind) which is at estimated [0.6 %] It is not simple. From here the idea of our project begins where it
aims. the aims of project of Blind assistance is promoting a widely challenge in computer vision such as
recognition of objects of the surrounding objects practiced by the blind on a daily basis.

1. Independence: Object detection and recognition empower blind individuals to move around
more freely and independently, without constantly relying on assistance from others.

2. Safety: Being able to detect obstacles and hazards in their path helps prevent accidents and
injuries, allowing blind individuals to navigate with greater confidence and security.

3. Efficiency: Recognizing objects enables blind people to locate and interact with items they
need more efficiently, whether it's finding a chair in a room or identifying a specific product on
a shelf.

4. Inclusivity: Access to object detection and recognition technology promotes inclusivity by

leveling the playing field and reducing barriers for blind individuals in various aspects of life,
such as education, employment, and social interactions.

5. Empowerment: Having the ability to independently identify and interact with objects fosters a
sense of empowerment and self-reliance among blind individuals, enhancing their overall well-
being and confidence.

4
 4. Literature Survey
Many researchers have contributed to this field. Various combinations of existing technologies have been
used. Braille systems, screen magnifiers, etc. went through some developments but later faced technical
issues.

1. Ayat A. Nada, was proposed, Stick solution use different technologies like infrared, ultrasonic sensor
and laser but they still have drawbacks. In the present study we introduce, light pressure, low-cost,
adaptable, fast response and low power utilization. Smart stick based infrared technology. A
combination of infrared sensors can reveal stair-cases and other obstacle presence in the user path,
within a range of two meters. The tentative results carry out good accuracy and the stick is able to
identify all of disincentives.

2. S. Innet, N.Ritnoom was proposed that blind people use a white stick as a tool for directing them
when they move or walk. In spite of, the white stick is helpful, it cannot give a high assurance that it
can assure blind people away from all level of hurdles. Several researchers have been obessed in
establishing electronic devices to protect blind people away from obstacles with a higher guarantee.
This study introduces an hurdles restraint alternative by using an electronic stick that serves as a tool
for blind people in walking. It exploits an infrared sensor for detecting hurdles along the roadway.
With all level of hurdles, the infrared stick facilitates to identify all type of earthly available in the
course such as concrete, wood, metal, glass, and human being. The outcome also shows that the stick
detects obstacles in range of 80 cm which is the same as the length of white stick. The twig is designed
to be small and light, so that blind people can carry it comfortably.

3. Ross Girshick , we propose a Fast Region-based Convolutional Network method (Fast R-CNN) for
object detection. Fast R-CNN frames on previous work to accurately distribute object proposals using
deep convolutional networks. Correlated to previous work, Fast R-CNN uses several innovations to
improve training and testing speed while also increasing detection accuracy. Fast R-CNN tracks the
very deep VGG16 network 9x faster than R-CNN, is 213x faster at test-time, and achieves a higher
mAP on PASCAL VOC 2012. Compared to SPP net, Fast R-CNN trains VGG16 3x faster, tests 10x
faster, and is more accurate. Fast R-CNN is implemented in Python and C++.

4. Multiple Distance Sensors Based Smart Stick for Visually Impaired People :Amit Kumar proposed
this system. In this system a novel low-cost yet durable and accurate smart stick to assist visually
impaired people while they walk in indoor/outdoor unstructured environments. There is a large group
of people who have difficulties in their daily routine work due to losing their eyesight. Walking with
confidence is one of them which may have different challenges in different environments/countries.
We have considered the Indian context where outdoor environments are often clustered and noisy.
Keeping these challenges in mind, a new smart stick is developed which is capable of detecting
obstacles of any height in front or slightly sideways of the person. The stick gives a fair idea about
the distance and the location of obstacles through vibration in hand and audio in the ear of the person.
The wireless connection has been set up using Bluetooth between theearphone and the stick. Different
5
 frequencies of the generated vibration and different tracks of the audio alert the person about the
distance of the obstacle. Real-time experiments have been conducted in different environments by
different people to observe the accuracy of the stick and results are quite encouraging.

5. In this system, an electronic aid to visually impaired people is designed which helps them to voyage
to the destination like normal people. The aiding system is built into a walking stick that shall be
carried by a visually impaired person. The aiding system acts like a reproduction vision. Sensors with
most accurate outputs are used in this work. The intelligent algorithm is used in the software so that
it is more user-friendly. A suitable walking stick is designed with all the stuff built-in. The canopy
people will able to budge from one place to another lacking other help. If such a system is developed,
it will act as a basic stand for the invention of more such devices for the canopy people in the potential
which will be cost-effective. And as far as the localization is anxious it will be able to provide accurate
information on the position of the canopy if in case they lost with help from the GPS. It will be a real
boon for the blind. The developed prototype gives good results in detecting obstacles paced at
distance in front of the user.These works report the designing of the multi-sensor blind stick. This
will be useful for visually impaired peoples. The canopy stick consists of tricky features which detect
obverse and top part of the obstacles, water stagnated/manholes on the ground. Due to these features
it is the best tool for blind and visually impaired people for on foot on the road. It is unforced, cost-
effective, configurable and simple to handle smart supervision systems. The system is planned
implemented, tested and verified. The results indicate that the structure is expert and inimitable in its
impending in specifying the source and space of the obstacles.

6. Navigation assistance for visually impaired (NAVI) refers to systems that can assist or guide people
with vision loss, ranging from partially sighted to blind, using sound commands. In this paper, a new
system for NAVI is presented based on visual and range information. Instead of using several sensors,
we choose one device, a consumer RGB-D camera, and take advantage of both range and visual
information. In particular, the main contribution is the combination of depth information with image
intensities, resulting in the robust expansion of the range-based floor segmentation. On one hand,
depth information, which is reliable but limited to a short-range, is enhanced with the long-range
visual information. On the other hand, the difficult and prone-to-error image processing is eased and
improved with depth information. The proposed system detects and classifies the main structural
elements of the scene providing the user with obstacle-free paths to navigate safely across unknown
scenarios. The proposed system has been tested on a wide variety of scenarios and data sets, giving
successful results and showing that the system is robust and works in challenging indoor
environments.

7. Laviniu Tepelea, Loan Gavrilut & Alexandru Gacsadi proposed the assistance system. In the system
toassist people with visual impairments, the smartphone proves to be very useful, but it requires
sensory modules external to the phone to detect obstacles and find a safe way. The Android
application we have made, offers not only a travel guide, but also other daily assistive functions such
as reading a poster or article, making phone calls, findingthe date, time, and battery level. The special
interface created for the blind has proved its efficiency, and the communication of the relevant
information verbally transmitted through the TTS to the earphones to one ear leads to a correct

6
understanding of the message and leaves the user the opportunity to receive other useful information
from the environment. External platforms communicate data from sensors to the phone via Bluetooth
and Wi-Fi where sensorial data fusion is made, and at the appropriate time, the relevant information
is communicated to the user, warning of the existence of an obstacle at a certain level. The accelerator
sensor can detect when the person with visual impairment is falling, and a phone call to a favorite
number is made, for example, the emergency service, and the light sensor detects the need to move
from outdoor guidance to indoor guidance and vice versa. Experimental tests made with the assistive
system have proven its usefulness, but they have also revealed that further testing is needed to find
the optimum obstacle detection distance, both inside buildings and in the outdoor environment. In the
future, more powerful and cheaper smartphones will be made, which will lead to more effective
assistance. Finally, this aiding system for visually impaired, based on a smartphone, but also using
other small external sensory modules, proves to be a viable, portable, low-cost, small-scale solution.
More important, it does not require many hours of training.

7
 5. Problem Statement

To make an efficient use of the system. Provide solution with least hardware requirement. To
develop an application that is cost efficient. Easy to use and accurate so that Visually Impaired People
can adopt the application quickly. The major challenge with visually impaired people is difficulty in
recognizing of objects. There are various issues they have to deal with, while performing various
daily tasks. They are unable to recognize objects while performing day-to-day activities, depriving
them from normal social life. One of the major problems faced by Blind people is detecting and
recognizing an obstacle in their path. To Implement application for Blind People. The projects
approach lies in developing a system based on OpenCV, which is capable of detect objects &
converting the text to speech and producing output in the form of audio signals to make the blind
person aware of the object in front of him. We choose OpenCV as our platform because it is a
standard representative of embedded device and is widely being used for devising low cost-system.

8
 6. Methodology and Proposed System Block Diagram

Methodology:

Object Detection: This system capture object from the camera and extracts the object from the object
with clear perception. With help of Text-to-Speech engine the text will be read for the user. Then It
normally converts a normal text into speech. Object detection is a computer technology related to
computer vision and image processing that deals with detecting the presence of objects with a limitative
box and types or classes of objects located in an image in digital images and videos. By Using object
detection, blind peoples can understand their surrounding environment without any challenges and
remain independent of others.

Object Recognition: Technology in the field of computer vision for finding and identifying objects in
an image or video sequence. Humans recognize a multitude of objects in images with little effort, despite
the fact that the image of the objects may vary somewhat in different viewpoints, in many different sizes
and scales or even when they are translated or rotated. Objects can even be recognized when they are
partially obstructed from view. This task is still a challenge for computer vision systems. Many
approaches to the task have been implemented over multiple decades.

Block Diagram:

Capture
Image

Using
MobileNetSSD
model

9
 7. Scope of the Project

Blind`s Vision Fist enables a visually impaired person to detect, recognize and track the object. Blind`s
Vision Fist also avoids the user from touching the hot object. Input given to Blind`s Vision Fist is a live
video feed with a frame rate of 60-70fps. The output from the system is given in the form of voice
assistance and vibration on the fist. Blind`s Vision Fist will navigate the user close to the object but the
task of picking the object depends on the user.

1. Environment Setup and Sensor Integration: Configure the hardware setup, such as cameras or
depth sensors, and integrate them with the OpenCV framework to capture real-time data from the
user's surroundings.

2. Object Detection Algorithm Development: Develop object detection algorithms using OpenCV
libraries to identify objects within the captured images or video streams. This may involve
techniques such as Haar cascades, Histogram of Oriented Gradients (HOG), or deep learning-based
approaches like YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

3. Object Recognition and Classification: Implement object recognition and classification

algorithms to categorize detected objects into different classes (e.g., chairs, doors, stairs) using
OpenCV's image processing and machine learning functionalities. This could involve training
classifiers on labeled datasets or using pre-trained models available within OpenCV or other
libraries like TensorFlow or PyTorch.

4. User Interface Design: Design an accessible user interface that presents the detected objects to the
blind user in a comprehensible manner. This may include auditory feedback, speech synthesis, or
tactile interfaces to convey information about the recognized objects.

5. Integration with Assistive Technologies: Explore integration possibilities with other assistive
technologies commonly used by blind individuals, such as screen readers or wearable devices, to
provide seamless interaction and navigation assistance.

6. Testing and Evaluation: Conduct rigorous testing and evaluation of the system's performance,
accuracy, and usability in real-world scenarios involving blind users. Gather feedback from users
to identify areas for improvement and iterate on the design accordingly.

7. Accessibility Features: Ensure that the system meets accessibility standards and includes features
such as keyboard navigation, high contrast display options, and compatibility with screen readers
to accommodate users with varying needs and preferences.

8. Documentation and Deployment: Create comprehensive documentation, tutorials, and user guides
to facilitate the deployment and usage of the object detection and recognition system by blind
individuals. Package the system for distribution and provide support for installation and setup.

9. Ethical Considerations: Address ethical concerns related to data privacy, consent, and bias in the
object detection algorithms, ensuring that the system respects the rights and dignity of blind users
while providing accurate and reliable assistance.

10
 8. Requirement Analysis

FUNCTIONAL REQUIREMENTS:
• Object Detection:
1. REQ-1: This system should capture object from the camera and extracts the object from the
object with clear perception.
2. REQ-2: The system should convert text to speech with help of Text-to-Speech engine and the
text should be read for the user.
3. REQ-3: It normally converts a normal text into speech.
4. REQ-4: Image processing should be done with the object present in front of camera
5. REQ-5: Convert the information to voice using headphone
6. REQ-6: Connect the web camera with system
7. REQ-7: Programming the system using Python language which will be powerful for processing.

• Object Recognition:
1. REQ-1: The system should process and analyze the camera records using system in real time.
2. REQ-2: The object should get detected and recognize the type of object in front of the blind.
3. REQ-3: Design and build an alarm system to notify the user about the recognized objects using
voice messages.

NON-FUNCTIONAL REQUIREMENTS:

a) Performance requirements:
1. The system gives advice or alerts user immediately.
2. The System gives accurate results.
3. Interactive, minimal delays, safe info transmission

b) Safety requirements:
1. Nobody will be harm while developing the system.
2. Easy to use.
3. System embedded with management procedures and validation procedures

c) Security requirements:
1. The system keeps all Users information’s with high security.
2. Identify all user, authenticate/verify user credentials, authorize user/third party, audit for
user’s usability, backup, server clustering, system policies

11
 9. Software Requirement Specification (SRS)

Hardware:
1. Processor – i3
2. Hard Disk – 5 GB
3. Memory – 1GB RAM
4. Earphone
5. Web Camera

Software:
1.Operating System: Windows XP and later versions
2. Front & Back End: HTML, CSS.
3.Programming Language: Python.
4.Database: MySQL/Firebase.
5.Libraries: OpenCV, pip, imutils, numpy

12
 10. Data Flow Diagram (DFD)

DFD level 0:

Fig: DFD Level 0

DFD level 1:

Fig: DFD Level 1

13
DFD level 2:

SYSTEM WEBCAM

WEBCAM

Fig: DFD Level 2

14
 11. UML Diagrams

A diagram is the graphical presentation of a set of elements, most often rendered as a connected graph
of vertices and arcs. you draw diagram to visualize a system from different perspective, so a diagram is
a projection into a system. For all but most trivial systems, a diagram represents an elided view of the
elements that make up a system. The same element may appear in all diagrams, only a few diagrams, or
in no diagrams at all. In theory, a diagram may contain any combination of things and relationships. In
practice, however, a small number of common combinations arise, which are consistent with the five
most useful views that comprise the architecture of a software-intensive system. For this reason, the UML
includes nine such diagrams:

1. Class Diagram
2. Activity Diagram
3. Use case Diagram
4. Sequence Diagram

Advantages of UML Diagrams:

• It’s Flexible & Well Known
• Effective Communication of the Software Architecture is Essential
• UML Is Easy to Understand
• Readability and Re-usability of UML Tools
• Serves as a Visual Representation between Classes and Entities
• Helps to Plan a Program before the Programming Takes Place

Disadvantages of UML Diagrams:

• Diagrams Can Get Overwhelming
• Too Much Emphasis on Design
• Sometimes unclear
• Time consuming

15
Class Diagram:

A Class is a category or group of things that has similar attributes and common behavior. A Rectangle is
the icon that represents the class it is divided into three areas. The upper most area contains the name,
the middle; area contains the attributes and the lowest areas show the operations. Class diagrams provides
the representation that developers work from. Class diagrams help on the analysis side, too Class is
nothing but a structure that contains both variables and methods. The Class Diagram shows a set of
classes, interfaces, and collaborations and their relating ships. There is most common diagram in
modeling the object oriented systems and are used to give the static view of a system. It shows the
dependency between the classes that can be used in our system. The interactions between the modules or
classes of our projects are shown below. Each block contains Class Name, Variables and Methods. The
Blind People Application Class diagram shows a basic class diagram. The ‘Capture’ class and the ‘Image
Proc’ class both inherit from the more general class, ‘R-Pi Camera’. The inheritance is shown using the
blank-headed arrow. The other class in the diagram is the ‘people’ class. The diagram is quite self-
explanatory and it clearly shows the different classes and how they are interrelated.

WEBCAM

Class diagram

16
Activity Diagram:

Activity diagram are the flowcharts, showing the flow of control from activity to activity. Activity
diagrams represent the business and operational work flows of a system. Activity Diagrams describe
how activities are coordinated to provide a service which can be at different levels of abstraction.
Typically, an event needs to be achieved by some operations, particularly where the operation is
intended to achieve a number of different things that require coordination, or how the events in a single
use case relate to one another, in particular, use cases where activities may overlap and require
coordination. It is also suitable for modeling how a collection of use cases coordinate to represent
business workflows. As shown in below diagram flow of system, first capture the image through R-Pi
camera And Send to system then Detect Object and Speech output with object Name.

WEBCAM

Activity diagram

17
Use Case Diagram:

A use case diagram in the Unified Modeling Language (UML) is a type of behavioral diagram defined
by and created from a use-case analysis. Its purpose is to present a graphical overview of the functionality
provided by a system in terms of actors, their goals (represented as usecases),and any dependencies
between those use cases. Use case diagram consists of use cases and actors and shows the interaction
between the use case and actors
• The purpose is to show the interactions between the use case and actor.
• To represent the system requirements from user’s perspective.
• An actor could be the end-user of the system or an external system
As shown in figure system can connect to R-PI camera, Ultrasonic sensor, Ear phone. The webcam
captures image from a given picture and the text from the captured image is converted into an audio
output using Raspberry Pi. The quality of the image captured will be high so as to have fast and clear
recognition due to the high-resolution camera. And also we measure the distance of the object using the
ultrasonic sensor. Both outputs are heard in an audio. The output device can be a headset connected to
the raspberry pi which can spell out the text document loud. The proposed system detects the obstacle
images which are present in outdoor and indoor with the help of a camera.

SOFTWARE KIT

SYSTEM

WEBCAM

Use case diagram

18
Sequence Diagram:

A Sequence Diagram is an interaction diagram that emphasis the time ordering of messages; a
collaboration diagram is an interaction diagram that emphasizes the structural organization of the objects
that send and receive messages. Sequence diagrams and collaboration diagrams are isomorphic, meaning
that you can take one and transform it into the other. Sequence diagram and collaboration diagram are
called INTERACTION DIAGRAMS. An interaction diagram shows an interaction, consisting of set of
objects and their relationship including the messages that may be dispatched among them. A sequence
diagram is an introduction that empathizes the time ordering of messages. Graphically a sequence
diagram is a table that shows objects arranged along the X-axis and messages ordered in increasing time
along the Y-axis. A sequence diagram for Blind People Helping hand represent communication flow
between People, System, Application. The system consists of a webcam interfaced with Raspberry-Pi ,
Pi Camera detects and find the type of object with the help of ultrasonic sensor. Ultrasonic sensor to
detect the real time hurdles while walking on the roads. The ultrasonic sensor used in this project plays
a vital role. It detects the object in front of this. When object is detected a indication sound is given to
the user via earphone.

WEBCAM

SYSTEM()

WEBCAM

Sequence diagram

19
 12. Implementation

# USAGE
# python real_time_object_detection.py --prototxt MobileNetSSD_deploy.prototxt.txt --model MobileNetSSD_deploy.caffemodel

# import the necessary packages

from imutils.video import VideoStream
from imutils.video import FPS
import numpy as np
import argparse
import imutils
import time
import cv2

# construct the argument parse and parse the arguments

ap = argparse.ArgumentParser()
#ap.add_argument("url")
#downloadData(args.url)
#ap.add_argument("-p", "--prototxt", default="MobileNetSSD_deploy.prototxt", required=True,
# help="path to Caffe 'deploy' prototxt file"
# "MobileNetSSD_deploy.prototxt for Caffe model or "
# )
#ap.add_argument("-m", "--model", default="MobileNetSSD_deploy.caffemodel", required=True,
# help="path to Caffe pre-trained model"
# "MobileNetSSD_deploy.caffemodel for Caffe model or "
# )
#ap.add_argument("-c", "--confidence", type=float, default=0.2,
# help="minimum probability to filter weak detections")
#args = vars(ap.parse_args())

ap.add_argument("--video", help="path to video file. If empty, camera's stream will be used")

ap.add_argument("--prototxt", default="./MobileNetSSD_deploy.prototxt.txt",
help='Path to text network file: '
'MobileNetSSD_deploy.prototxt for Caffe model or '
)
ap.add_argument("--model", default="./MobileNetSSD_deploy.caffemodel",
help='Path to weights: '
'MobileNetSSD_deploy.caffemodel for Caffe model or '
)
ap.add_argument("--confidence", default=0.2, type=float, help="confidence threshold to filter out weak detections")
args = vars(ap.parse_args())

# initialize the list of class labels MobileNet SSD was trained to

# detect, then generate a set of bounding box colors for each class
CLASSES = ["background", "aeroplane", "bicycle", "bird", "boat",
"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
"sofa", "train", "tvmonitor"]
COLORS = np.random.uniform(0, 255, size=(len(CLASSES), 3))

# load our serialized model from disk

print("[INFO] loading model...")
net = cv2.dnn.readNetFromCaffe(args["prototxt"], args["model"])
#net = cv2.dnn.readNetFromCaffe(args.prototxt, args.model)

# initialize the video stream, allow the cammera sensor to warmup,

# and initialize the FPS counter
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
time.sleep(2.0)
fps = FPS().start()

# loop over the frames from the video stream

20
while True:
# grab the frame from the threaded video stream and resize it
# to have a maximum width of 400 pixels
frame = vs.read()
frame = imutils.resize(frame, width=400)

# grab the frame dimensions and convert it to a blob

(h, w) = frame.shape[:2]
blob = cv2.dnn.blobFromImage(cv2.resize(frame, (300, 300)),
0.007843, (300, 300), 127.5)

# pass the blob through the network and obtain the detections and
# predictions
net.setInput(blob)
detections = net.forward()

# loop over the detections

for i in np.arange(0, detections.shape[2]):
# extract the confidence (i.e., probability) associated with
# the prediction
confidence = detections[0, 0, i, 2]

# filter out weak detections by ensuring the `confidence` is

# greater than the minimum confidence
if confidence > args["confidence"]:
# extract the index of the class label from the
# `detections`, then compute the (x, y)-coordinates of
# the bounding box for the object
idx = int(detections[0, 0, i, 1])
box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
(startX, startY, endX, endY) = box.astype("int")

# draw the prediction on the frame

label = "{}: {:.2f}%".format(CLASSES[idx],
confidence * 100)
cv2.rectangle(frame, (startX, startY), (endX, endY),
COLORS[idx], 2)
y = startY - 15 if startY - 15 > 15 else startY + 15
cv2.putText(frame, label, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, COLORS[idx], 2)

# show the output frame

cv2.imshow("Frame", frame)
key = cv2.waitKey(1) & 0xFF

# if the `q` key was pressed, break from the loop

if key == ord("q"):
break

# update the FPS counter

fps.update()

# stop the timer and display FPS information

fps.stop()
print("[INFO] elapsed time: {:.2f}".format(fps.elapsed()))
print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))

# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()

21
• OpenCV algorithm:

OpenCV, short for Open Source Computer Vision Library, is a powerful open-source computer vision
and machine learning software library. It provides various tools and functions that enable developers to
perform tasks such as image processing, object detection, feature extraction, and more.
For object detection specifically, OpenCV offers several pre-trained models such as Haar cascades and
deep learning-based models like Single Shot Multibox Detector (SSD), You Only Look Once (YOLO),
and Faster R-CNN. These models allow users to detect objects within images or videos with high
accuracy.

OpenCV's object detection capabilities are widely used in various applications including surveillance,
autonomous vehicles, facial recognition, and industrial automation. Its versatility, combined with a large
community and extensive documentation, makes it a popular choice for developers working on computer
vision projects.

22
• MobileNetSSD- Model

MobileNet-SSD is a lightweight deep learning model that combines the MobileNet architecture with the
Single Shot Multibox Detector (SSD) framework for efficient object detection and recognition on mobile
and embedded devices. MobileNet utilizes depthwise separable convolutions to reduce computational
complexity while maintaining accuracy, making it suitable for deployment on devices with limited
resources. SSD enables the model to predict bounding boxes and class probabilities directly from feature
maps, allowing for fast and accurate object localization within images.

MobileNet-SSD is trained to detect and recognize a wide range of objects, making it applicable in
various domains such as mobile augmented reality, surveillance, and autonomous systems. Its compact
design and high performance make it ideal for real-time object detection tasks in resource-constrained
environments.

23
 13. Results and Test Cases

Results:

24
Test Cases:

Test Test Case Input Data Expected Result Actual Result Status
Case ID Description
TC_01 Checking that Install System should be System is installed Pass
system works Python in the installed and system and system is
properly. system should work as working as
intended intended
TC_02 Checking that Web Connect Web Camera should get Camera gets Pass
Camera works Camera to connected and connected and
properly. system should turn on when turns on when
system is turned on system is turned on
TC_03 Installing libraries Import Libraries needed for Libraries needed Pass
needed for video libraries video streaming for video streaming
stream should get installed are installed
TC_04 Detecting the Place object Object should be Object is detected Pass
object in front of the detected and voice and voice output is
camera output should be given
given
TC_05 Recognizing the Place object Object should be Object is detected Pass
type of the object in front of the detected and type of and type of object
camera object should get is recognized and
recognized also also speech output
speech output should is given
be given
TC_06 Detecting and Place an Object at a distant Object at a distant Pass
recognizing the object at level should get level got detected
object at a distant distant level detected and and recognized by
level from the recognized by the the Web-Camera
camera Web-Camera

TC_07 Checking the Place object The object placed in The object placed Pass
accuracy level of in front of a the front of the in the front of the
the detected object camera camera should get camera is detected
detected accurately accurately

TC_08 Testing the Place the The object placed in The object placed Pass
detection of the object in lighting camera in lighting camera
object by placing lighting should be detected is detected with
in lighting condition with good accuracy good accuracy
conditions level by the camera level by the camera
and also speech and also speech
output should be output is given
given

25
TC_09 Testing the Place the
The object placed in The object placed Pass
detection of the object with aa different in a different
object by placing different orientation should be orientation is
an object in a orientation detected and detected and
different recognized by the recognized by the
orientation camera camera
TC_10 Checking the Place All the objects All the objects Pass
response time of multiple placed sequentially placed sequentially
the proposed objects in front of the camera in front of the
system to detect sequentially should be detected camera are
and recognize the in front of and recognized detected and
object camera accurately recognized
accurately

26
 14. Challenges Faced

1. Limited Dataset Availability: Obtaining diverse and representative datasets of objects

encountered in various environments (indoor, outdoor, public spaces) poses a challenge.
Adequate training data are essential for developing accurate object detection and recognition
models.

2. Real-Time Processing: Processing images or video streams in real-time to detect and recognize
objects requires efficient algorithms and optimizations to ensure timely feedback to the user.
OpenCV's performance on different hardware configurations must be considered to achieve real-
time processing.

3. Accuracy and Robustness: Achieving high accuracy and robustness in object detection and
recognition, especially in complex and dynamic environments, is challenging. Variations in
lighting conditions, occlusions, object orientations, and cluttered backgrounds can affect the
performance of detection algorithms.

4. Hardware Limitations: Some hardware platforms commonly used by blind individuals, such
as smartphones or low-power embedded systems, may have limited processing capabilities.
Optimizing the object detection and recognition algorithms to run efficiently on these platforms
while maintaining accuracy is a challenge.

5. User Interface Design: Designing an intuitive and accessible user interface for blind users to
interact with the object detection and recognition system poses challenges. Ensuring that the
interface provides meaningful feedback about detected objects in a non-visual manner, such as
through auditory or haptic feedback, requires careful consideration.

6. Integration with Assistive Technologies: Integrating the object detection and recognition
system with other assistive technologies commonly used by blind individuals, such as screen
readers or wearable devices, can be challenging. Ensuring seamless interaction and compatibility
with existing assistive technologies requires careful coordination and testing.

7. Ethical Considerations: Addressing ethical concerns related to data privacy, consent, and bias
in the object detection algorithms is crucial. Ensuring that the system respects the privacy and
autonomy of blind users while providing accurate and unbiased assistance is essential.

27
 15. Advantages and Disadvantages

Advantages:

1. Access to authorized personnel only.

2. No additional devices need to be carried for object detection.
3. The system is used in the real time object detection.
4. It helps to Blind person to prevent from dangerous location.
5. This project is to provide cost effective way to allow path planning for blind people.
6. Reducing the bulkiness and making the system portable for the application to come into picture in
real life.
7. Modifying speed, accuracy, and reliability of the system
8. Making the Raspberry Pi speak and respond to instructions given by blind person (eg.
Navigation)

Disadvantages:
1. Slow Internet Connection.
2. Low Light Intensity affects accuracy of model

28
 Future Scope

In the domain of Artificial Intelligence, there was only the capturing the object using R-Pi in the existing
system. At present, the work was successful to detect, recognize and track the object. And is used to
avoids the user from touching the hot object. To further this project can be followed out with any other
advanced devices by using simple coding language to get it less complicated. The complication can be
reduced by a tiny gadget which could be more useful those people in this electronic world. The future
perspective of this project is to increase the object recognition to provide an exact distance measurement
between the people and object. However, for developing an application that involves many objects that
are fast-moving, you should instead consider faster hardware.

29
 Conclusion

In this project, we have implemented an image to speech conversion technique using a raspberry pi. This
system is much helpful Application for blind people. This system will be very easy to use. This
Application is very helpful towards the blind people and makes very easy for them to use electronic
gadgets with (text to speech) TTS technology, they will be able to interact more efficiently to the
electronic system. This is an economical as well as an efficient device for the visually impaired people.
The device is compact and helpful to the society. The proposed system is mobile, robust, and efficient.
Also, it creates a virtual environment and this system provides a sense of assurance as it voices the name
of the object recognized. The usage of OpenCV has optimized the system and brought down the cost
drastically compared to conventional systems. The proposed system is simple in design, easy to install
and is reliable to a great extent. Blind person is able to detect and recognize the object in front (within a
specific range) through audio output. In all, the developed system is able to develop a technical approach
for solving a real- life problem in a time and cost-effective manner to a great extent.

30
 References

Reference Papers: (Sample Format)

1. Ayat A. Nada Department of Computers and Systems Electronics Research Institute, Giza, Egypt ,
“Assistive Infrared Sensor Based Smart Stick for Blind People” [email protected]

2. Arnesh Sen Kaustav Sen Jayoti Das Jadavpur University: Dept. of Physics, “Ultrasonic Blind Stick
For Completely Blind People To Avoid Any Kind Of Obstacles”, Kolkata, India
[email protected].

3. “An Application of Infrared Sensors for Electronic White Stick” S. Innet 1, N. Ritnoom
21Department of Computer and Multimedia Engineering 2Department of Electrical Engineering
University of the Thai Chamber of Commerce.

4. Sharang Sharma, Manind Gupta, Amit Kumar, Meenakshi Tripathi, Manoj Singh Gaur, “Multiple
Distance Sensors Based Smart Stick for Visually Impaired People.”, 2017.

5. M.Micheal Priyanka, M.Michael Dharsana, “Navigational Aiding System For Visually Impaired",
Third International Conference On Science Technology Engineering And Management
(ICONSTEM), 2017

6. A. Aladrén, G. López-Nicolás, Luis Puig, and Josechu J. Guerrero, "Navigation Assistance for the
Visually Impaired Using RGB-D Sensor With Range Expansion." IEEE Systems Journal 2014.

7. Laviniu _epelea, Ioan Gavrilu_, Alexandru Gacsádi, “Smartphone Application to Assist Visually
Impaired People”, 14th International Conference on Engineering of Modern Electric Systems
(EMES), 2017.

8. Kabalan Chaccour and Georges Badr," Novel indoor navigation system for Visually Impaired and
blind people",2015.

9. Jinqiang Bai, Shiguo Lian, Zhaoxiang Liu, Kai Wang, and Dijun Liu, "Smart Guiding Glasses for
Visually Impaired People in Indoor Environment", IEEE Transactions on Consumer Electronics, Vol.
63, No. 3, August 2017,pp.258-266

10. Zoran Zivkovic. Improved Adaptive Gaussian Mixture Model for Background Subtraction Pattern
Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on 20 Sep 2004

31
Reference Websites:
➢ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5066058/
➢ https://ieeexplore.ieee.org/document/8982222
➢ https://arxiv.org/abs/1506.02640
➢ https://aws.amazon.com/rekognition/
➢ https://www.sciencedirect.com/science/article/pii/S136481521730246X
➢ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5730768/
➢ https://www.ericsson.com/en/blog/2016/10/firmware-updates-for-wearables
➢ https://ieeexplore.ieee.org/document/8466307

32