Real Time Object Detection System using Deep Learning

KAVURI VANDHAN RANGINENI ARJUN BIJU

ADHIAKRI SAIKIRAN
MURALI ROHITH Student MAHADEV
Student
Student Student Dept. of CSE Professor
Dept. of CSE
Dept. of CSE Dept. of CSE Apex Institute of Dept. of CSE
Apex Institute of
Apex Institute of Apex Institute of Technology Apex Institute of Technology
Technology
Technology Technology Chandigarh Chandigarh University
Chandigarh University
Chandigarh University Chandigarh University University Mohali, Punjab, India
Mohali, Punjab, India
Mohali, Punjab, India Mohali, Punjab, India Mohali, Punjab, India [email protected]
[email protected]
[email protected] [email protected] [email protected]

Abstract— Applications for real-time object deep learning algorithms have transformed the field of
detection in computer vision range from autonomous object detection. In this project, we focus on real-time object
driving to video surveillance. When it comes to object detection using the Single Shot MultiBox Detector (SSD)
detection, deep learning techniques have demonstrated algorithm. SSD stands out among other object detection
great success by offering high accuracy and efficiency. In methods due to its impressive balance.
this study, we employ the Single Shot MultiBox Detector
(SSD) algorithm to investigate real-time object detection. Between accuracy and computational efficiency. It is
We investigate SSD's core ideas, network structure, and particularly suited for real-time since it has the
training process for real-time object detection. The benefit of immediately predicting object bounding
research dives into the distinctive features of SSD, such as boxes and class labels in a single run. The major goal
the application of default boxes and multiscale feature of this project is to create a real-time object detection system
maps for effective and precise identification. On that is reliable and effective, based on the SSD algorithm.
benchmark datasets, we measure parameters like mean We want to investigate SSD's core ideas, network structure,
average precision (mAP), detection speed, and training approach, as well as special elements like multi-
computational efficiency to determine how well the SSD scale feature maps and default boxes. We want to
algorithm performs. The usefulness of SSD for real-time accomplish precise and quick object detection in a variety of
object detection across many different object types is settings by taking advantage of the SSD algorithm's
demonstrated by experimental findings. Furthermore, we advantages.
examine potential SSD algorithm improvements and
adjustments to overcome its drawbacks and boost This study also examines the SSD-based object detection
performance in particular cases. The results of this study system's evaluation and performance analysis. To assure
offer knowledge to researchers and professionals in the real-time capabilities, we check its accuracy using measures
field and help enhance our understanding of real-time like mean average precision (mAP) and assess its speed and
object detection utilizing the SSD method. computing efficiency. In addition, we investigate potential
improvements and adjustments to the SSD algorithm to
Keywords—real-time object detection, deep
overcome its drawbacks and boost performance in particular
learning, Single Shot MultiBox Detector (SSD), computer
circumstances.
vision, autonomous driving, video surveillance, accuracy,
efficiency, network architecture, training methodology. The SSD technique, developed by Liu et al. in 2016,
combines object localization and classification in a single
neural network architecture to overcome the difficulty of
I. INTRODUCTION real-time object identification. SSD directly predicts item
bounding boxes and class labels at various scales, in contrast
A key job in computer vision, real-time object identification to region-based methods that rely on region proposal
has several applications, such as autonomous driving, video techniques, such as selective search or region proposal
surveillance, and augmented reality. For systems to be networks. As a result, it strikes a fair balance between
responsive and intelligent, they must be able to recognize efficiency and detection precision, making it suitable for
things reliably and effectively in real time. Deep learning- real-time applications.
based algorithms have significantly outperformed
conventional methods in the field of object detection in We give a thorough analysis of real-time object detection
recent years. Due to its effectiveness and efficiency, the using the SSD algorithm in this research study. Our goal is
Single Shot MultiBox Detector (SSD) algorithm has to fully comprehend the SSD's guiding ideas, network
attracted a lot of attention among these methods. Numerous architecture, and training process to achieve precise and
computer vision applications, including robots, autonomous effective object identification in real-world scenarios. SSD is
driving, and surveillance systems, all heavily rely on real- created to capture objects at different scales and aspect
time object detection. Intelligent systems must be able to ratios, boosting identification performance by utilizing deep
identify objects reliably and effectively in real-time convolutional neural networks (CNNs) and cutting-edge
situations to be able to comprehend and interact with their methods including multi-scale feature maps and default
surroundings. By reaching astounding accuracy and speed, boxes.
We want to assess the SSD algorithm's performance using obtained high mAP scores while maintained quick inference
metrics like mean average precision (mAP), detection speed, times.
and computational efficiency on common benchmark
datasets like PASCAL VOC and COCO. We examine the Although SSD has shown to be a reliable and effective
benefits and drawbacks of SSD through comprehensive method, it does have some restrictions. Due to the fixed-size
tests, as well as its suitability for real-time object recognition default boxes, it has a limited capacity to detect small things
jobs involving a variety of object classifications. accurately. Researchers have looked into methods like
This study provides a thorough examination of real-time anchor optimization, which adaptively modifies the default
object detection utilizing the SSD method. The conclusions box scaling and aspect ratios depending on the dataset
and insights offered here can help researchers and properties, to get around this. To improve the contextual
practitioners comprehend and make use of SSD's information and capture more accurate object boundaries,
capabilities, stimulating improvements in real-time object feature fusion techniques and context modelling have also
identification and making it possible to create intelligent been suggested. Additionally, studies on the music
systems that depend on precise and effective object recommendation system have been conducted. According to
recognition. one such research, a preliminary approach to Hindi music
II. RELATED WORK mood classification has been described, that exploits simple
features extracted from the audio.
A. Literature Survey
Deep learning-based real-time object detection has made MIREX (Music Information Retrieval Evaluation exchange)
considerable strides in recent years, with many algorithms mood taxonomy gave an average accuracy of 51.56% using
exhibiting astounding accuracy and speed. In this literature the 10-fold cross validation. This is in addition to an article
review, we explicitly use the Single Shot MultiBox Detector that claims that the current music recommendation study is
(SSD) technique to explore the state of the art in real-time based on Deep learning model accuracy and effectiveness
object detection. Deep learning-based object identification have recently been the main areas of focus in real-time
techniques have become popular because they can object detection. The feature pyramid network (FPN) and
automatically learn intricate features and patterns from data, SSD algorithm integration is one famous method. To detect
doing away with the necessity for feature engineering by objects of varying sizes, FPN uses a top-down architecture
hand. For real-time object detection, Liu et al.'s SSD that creates feature maps at various scales. The management
algorithm stands out as a practical and successful method. of scale fluctuations and improved detection performance,
particularly for small objects, have demonstrated
The SSD algorithm's architecture, which unifies feature encouraging results from this upgrade.
extraction and object detection into a single network, is the
key idea behind it. By using this strategy, the model can The investigation of innovative loss functions is another
predict object bounding boxes and class probabilities field of research in real-time object detection. By
simultaneously at various scales and aspect ratios. SSD introducing the focus loss, Lin et al. address the problem of
strikes an excellent mix between accuracy and performance class imbalance during training. Hard examples are given
by utilizing multi-scale feature maps and default boxes, more weight, highlighting their significance in the learning
making it suitable for real-time applications. In terms of process. Particularly for difficult item classes that are
network design, SSD uses a base network for feature underrepresented in the training data, this loss function has
extraction, such as VGG-16 or ResNet. Hierarchical visual shown to be effective in enhancing detection accuracy. The
feature representations, which are essential for object literature review concludes by highlighting the importance
detection, are captured by these deep convolutional neural of real-time object detection using deep learning and
networks (CNNs). concentrating on the Single Shot MultiBox Detector (SSD)
technique.
Additionally, SSD introduces auxiliary convolutional layers
at various scales to predict bounding box offsets and class
With a balance between precision and efficiency, SSD
probabilities. This multi-scale feature fusion enhances the
presents an appealing choice for real-time applications. A
detection performance, especially for objects of different
benchmark dataset review of SSD demonstrates its
sizes and aspect ratios.
competitive performance, and current research investigates
The SSD algorithm's performance in real-time object improvements and tweaks to overcome its drawbacks. The
detection has been evaluated on benchmark datasets like literature review establishes the context for the study by
PASCAL VOC and COCO. To test the algorithm's real-time giving readers a thorough overview of the current state of
capabilities, detection speed and computing efficiency are the field and highlighting the necessity of further
assessed along with the mean average precision (mAP) developments in real-time object detection using the SSD
metric, which is frequently used to gauge detection accuracy. algorithm.
In comparison to other well-known object detection
techniques, SSD has demonstrated competitive performance, B. Existing Systems
 Real-Time Object Detection for Medical SSDs handled challenging settings while
Imaging: For real-time object detection in medical maintaining high detection accuracy.
imaging, such as the identification of tumors or  Real-Time Object Detection for Autonomous
anatomical structures, this system used the SSD Vehicles: For real-time object detection in
algorithm. The method made efficient and accurate autonomous driving scenarios, our system used the
object detection possible by incorporating SSD into SSD algorithm. It addressed the issues raised by
medical imaging systems, which facilitated medical autonomous vehicles, such as the need to recognize
diagnosis and treatment planning. traffic signals, automobiles, and people. The system
 Real-Time Object Detection for Sports demonstrated how well SSD performs in terms of
Analytics: Targeting player tracking and action rapid and accurate object detection for safe driving.
recognition, this system used the SSD algorithm to
real-time object detection in sports analytics. The III. PROPOSED METHODOLOGY
technology enabled advanced analysis and insights
The research paper presents a detailed methodology for real-
in sports-related applications by accurately and
time object detection using the Single Shot MultiBox
quickly detecting athletes and their movements.
Detector (SSD) algorithm. The methodology encompasses
 Real-Time Object Detection in Unmanned Aerial
the key steps involved in training and deploying the SSD
Vehicles (UAVs): This system concentrated on
model for accurate and efficient real-time object detection.
real-time object detection in unmanned aerial
The following sections outline the methodology in a step-by-
vehicles (UAVs) or drones utilizing the SSD
step manner:
algorithm. The goal was to make autonomous
UAVs capable of real-time obstacle detection and
A. Dataset Preparation:
avoidance, ensuring secure navigation and
operation in changing situations. Getting and getting ready the dataset for training and
 Real-Time Object Detection on Mobile Devices: evaluation is the initial stage. This involves choosing a
This approach aimed to bring the power of deep suitable dataset, such as PASCAL VOC or COCO, and
learning to platforms with limited resources by making sure that object bounding boxes and class labels are
focusing on real-time object detection on mobile properly annotated. Techniques for enhancing data, such as
devices. It put into practice an improved SSD random cropping, rotation, and flipping, can be used to
methodology that made use of hardware improve the dataset's diversity and robustness. The network
acceleration methods like GPU and CPU architecture is then established. A base network, such as
optimizations. The technology demonstrated how VGG-16 or ResNet, is selected and adapted for the SSD
real-time object detection on portable devices is framework. The underlying network is adjusted for the
possible. object detection job after being pretrained on a sizable
dataset like ImageNet.
C, Existing Algorithms/Tools
The SSD algorithm predicts item bounding boxes by using
 Real-Time Object Detection in Video Streams: default boxes or anchors at various scales and aspect ratios.
This system was designed to detect objects in video Statistical analysis or methods like k-means clustering are
streams in real-time and track them across multiple used to find the best arrangements for these default boxes.
frames. Fast and precise object detection was Multi-scale feature maps are extracted at multiple layers of
accomplished using the SSD method, allowing for the network architecture to capture objects of varied sizes.
real-time object tracking in dynamic settings. These feature maps are linked to various default box scales,
 Real-Time Object Detection for Robotics allowing the model to recognize objects at various
Applications: To enable real-time object detection resolutions and adapt to scale variations.
for robot perception and interaction, this system
The bounding box regression and classification components
integrated the SSD method into robotics
of the loss function are defined to train the SSD model. To
applications. The system showed how well SSD
solve certain issues like class imbalance or small item
performed various robot tasks like item
recognition, modifications or alterations to the loss function
manipulation and recognition by offering accurate
may be suggested. Using the prepared dataset, the SSD
and quick object detection.
model is optimized during the training phase.
 Real-Time Object Detection in Surveillance
Backpropagation and methods like stochastic gradient
Videos: To provide real-time object detection in
descent or adaptive learning rate approaches are used to
surveillance videos, this system used the SSD
update the model's parameters.
algorithm. The emphasis was on identifying and
following people as well as suspicious activity in The accuracy, localization, and recognition of objects are
crowded settings. The system showed off how well measured using evaluation metrics such mean average
precision (mAP), intersection over union (IoU), and
precision-recall curves. The experimental setup comprises accurately in real-world circumstances, carry out extensive
describing the hardware and software configurations used testing and validation.
for training and assessment, including the GPU or CPU
Analyze the system's efficiency in terms of resource use,
resources, software frameworks (such as TensorFlow,
detection speed, and accuracy. Considerations for
PyTorch), and other dependencies. The real-time object
Deployment and Deployment: Consider variables including
identification performance of the trained SSD model is
platform compatibility, system dependencies, the
assessed on a different validation or test dataset. It is
deployment environment, and scalability as you prepare the
possible to compare the model's detection efficiency, speed,
integrated system for deployment. Record any
and computational effectiveness with different SSD
recommendations or guidelines for implementing the system
variations or with existing techniques.
in practical circumstances. Updates & Continuous
B. Integration: Improvement: Keep an eye on how the integrated system is
performing, get user input, and make any necessary updates
Integration, which comprises integrating the real-time object
or improvements. This can entail updating the SSD model
detection system based on the SSD algorithm into the
with new information, optimizing hyperparameters, or
intended application or system, is a critical component of the
incorporating fresh methods as they emerge. The real-time
project. The following steps are often included in the
object detection system based on the SSD algorithm can be
integration process: System comprehension: Learn
effortlessly incorporated into the target application or system
everything there is to know about the intended system or
by adhering to these integration stages, enabling precise and
application where the real-time object detection system will
effective object recognition in real-time circumstances.
be incorporated.

Determine the system's precise requirements, limitations, IV. HARDWARE and SOFTWARE REQUIREMENTS
and goals. Processing of Input and Pre-processing: Choose a
A. Hardware Requirements
method for acquiring and pre-processing the input data so
that it is compatible with the SSD algorithm. Managing Computer or Server: A computer or server with sufficient
numerous data sources, such as photos, videos, or live processing power and memory is needed to train and deploy
camera feeds, may be required. Any required pre-processing the SSD model. The specifications of the computer/server
operations, such as resizing, normalization, or frame will depend on the size of the dataset, complexity of the
extraction, should be carried out. Integration of the SSD model, and the desired speed of object detection.
Model: Integrate the prepared SSD model into the intended
GPU (Graphics Processing Unit): Training and inference
program or system.
of deep learning models, such as SSD, can benefit
This entails loading the model's architecture and parameters significantly from GPU acceleration. A high-performance
and setting up the software framework or libraries required GPU with CUDA support is recommended to speed up the
to carry out the model inference. Real-Time Object training and inference processes.
Detection: Apply the SSD model to real-time object
Camera or Video Input: If the real-time object detection
detection. Feed the model the pre-processed input data to get
system is designed to work with live camera feeds or video
the expected bounding boxes and class labels for items that
streams, a compatible camera or video input device is
are detected. Post-processing and Visualization:
required.
Using post-processing techniques, such as non-maximum
B. Software Requirements
suppression (NMS) to get rid of overlapping bounding
boxes, you can improve the outcomes of the identified item Operating System: The choice of operating system depends
detection. Overlaying bounding boxes and class labels on on the specific software frameworks and libraries used for
the incoming data can help you see the discovered items. implementing the SSD algorithm. Common choices include
Integrate the information about identified objects with the Windows, Linux (e.g., Ubuntu), or macOS. Deep Learning
system's downstream duties or functions if the object Framework: The SSD algorithm needs to be implemented
detection system is a part of a bigger application. This could and trained using a deep learning framework. SSD
involve activities like user engagement, tracking, and implementations are already embedded into well-known
recognition. frameworks like TensorFlow, PyTorch, or Caffe, which also
provide the tools required for model training and
Performance Optimization: To ensure real-time performance
deployment.
and computational effectiveness, optimize the integration. To
accelerate the object detection process, this may entail CUDA and cuDNN: If GPU acceleration is utilized,
methods like model quantization, hardware acceleration (for installing CUDA (Compute Unified Device Architecture)
example, GPU utilization), or algorithmic optimizations. and cuDNN (CUDA Deep Neural Network) libraries is
Validation and Testing To guarantee that the integrated necessary to enable GPU support for deep learning
system satisfies the desired requirements and operates frameworks.
Additional Libraries: Depending on the specific Speed and Efficiency: By assessing the detecting speed, the
implementation and requirements, additional libraries and system's real-time component was evaluated. Even on
packages may be needed, such as NumPy, OpenCV (for systems with limited resources, real-time object detection
image and video processing), and matplotlib (for was possible thanks to the SSD algorithm's quick processing
visualization). speeds. Additionally, the algorithm's computational
effectiveness was assessed while accounting for the size of
Development Environment: An integrated
the model and memory requirements.
development environment (IDE) or text editor of choice,
such as PyCharm, Jupyter Notebook, or Visual Studio Code Scale and Variability: On objects with various scales and
can facilitate the coding and development process. aspect ratios, the SSD method was put to the test. The
program successfully handled scale changes, correctly
V. RESULTS and DISCUSSION
detecting both small and large objects, according to the
Accurately identifying human emotion or mood is results. The system's resilience was enhanced by the usage
challenging since each person has distinctive facial traits. of default boxes and multi-scale feature maps.
But it can be recognized to some extent with the right facial
Comparison with Existing Methods: The SSD algorithm's
expressions. The device's camera ought to have a greater
performance was compared to that of other cutting-edge
resolution. The following are some screenshots that were
object detecting techniques. The outcomes showed that
taken while using the Android application that we designed.
while maintaining real-time processing capabilities, the SSD
method offered competitive accuracy. The benefits of the
SSD method were emphasized, including its ease of use,
single-shot detection strategy, and multi-scale feature
representation.

Limitations and Challenges: The constraints and difficulties

faced when implementing the SSD method were also
discussed. The system might have trouble correctly detecting
items with extreme aspect ratios or objects that are obscured,
for instance. Future improvements were suggested for
methods to overcome these restrictions, such as data
augmentation techniques or changes to the design.

Practical Applications: The real-world uses of the SSD

A delegated region proposal network is not used by SSD. algorithm-based real-time object identification system were
Instead, it comes to a straightforward solution. It uses covered. These could include robotics, autonomous vehicles,
modest convolution filters to calculate the location and class or surveillance systems, all of which require quick and
scores. To produce predictions, SSD applies 3 3 convolution precise object identification. The system's potential
filters to each cell after extracting the feature maps. (These influence and advantages in real-world situations were
filters compute the outcomes in the same manner as standard emphasized.
CNN filters.) Each filter produces 25 channels total, with 21
scores per class and an additional boundary box. Future Directions: Insights into potential routes for future
research and changes to the SSD algorithm were also shared
Results: in the comment section. Advanced techniques like attention
mechanisms, unique training procedures, or tackling
Evaluation Metrics Using benchmark datasets, the SSD difficulties in object recognition, including handling
algorithm's effectiveness for real-time object detection was crowded environments or identifying things in difficult
assessed. The accuracy of object detection was measured environmental conditions, are all potential areas for
using metrics like mean average precision (mAP). It is also additional research.
possible to record additional metrics including precision,
The provided picture demonstrates the functioning of the
recall, and F1 score.
SSD algorithm, showcasing its ability to detect all objects
Detection Accuracy: Across a range of item types, the SSD present in the image.
algorithm achieved good detection accuracy. The system's
overall performance in terms of accurately localizing and VI. CONCLUSION
recognizing objects was represented by the mAP score.
Results revealed that when compared to other object Object recognition is one of deep learning's newest and most
detection techniques, the SSD algorithm produced intriguing fields. Nearly all smartphone cameras perform
competitive or cutting-edge performance. object detection, which is a common use. on numerous deep
learning models that can recognize and recognize objects in
real time. YOLOv3, Tiny-YOLOv3, and Faster R-CNN have networks and fully connected neural networks. In
been shown to be the most suited and effective deep learning 2017 International Conference on Energy,
models to perform real-time object identification and Communication, Data Analytics and Soft
recognition on scale engineering vehicles after examining Computing (ICECDS) (pp. 1485-1488). IEEE.
the performance of these algorithms on standard datasets. Redmon, J., Divvala, S., Girshick, R., & Farhadi,
A. (2016). You Only Look Once: Unified, Real-
It is determined that Faster R-CNN exhibits superior Time Object Detection. In Proceedings of the IEEE
accuracy than SSD and FCN models while performing as Conference on Computer Vision and Pattern
quickly as those models. As One of the most challenging, Recognition (CVPR) (pp. 779-788).
intricate, and important tasks in the field of computer vision Ren, S., He, K., Girshick, R., & Sun, J. (2016).
today is object detection and recognition. As far as we are Faster R-CNN: Towards Real-Time Object
aware, the main objective of this project was to record real- Detection with Region Proposal Networks. In
time items in pictures, videos, or webcams. Advances in Neural Information Processing
Future innovations can be concentrated by putting the Systems (NIPS) (pp. 91-99).
project on a system with GPU for quicker outcomes and Liu, W., Anguelov, D., Erhan, D., Szegedy, C.,
greater accuracy. Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD:
Single Shot MultiBox Detector. In European
 It can be improved and innovated in the future by
Conference on Computer Vision (ECCV) (pp. 21-
anyone without worrying about complexity.
37).
 MS COCO, for instance, detects small objects in a
Redmon, J., & Farhadi, A. (2018). YOLOv3: An
variety of face detection tasks and applications. for
Incremental Improvement. arXiv preprint
improved localization of tiny objects across partial
arXiv:1804.02767.
obstacles. We will make certain adjustments to the
Liu, W., Rabinovich, A., & Berg, A. C. (2015).
network design to successfully recognize small
ParseNet: Looking Wider to See Better. arXiv
objects with efficiency and accuracy, decreasing
preprint arXiv:1506.04579.
reliance on data networks.
Dai, J., Li, Y., He, K., & Sun, J. (2016). R-FCN:
Thus, it can be said in the end that to improve accuracy and Object Detection via Region-based Fully
performance, pre-processing techniques like edge detection Convolutional Networks. In Advances in Neural
and increasing image augmentation and contrast should be Information Processing Systems (NIPS) (pp. 379-
used. 387).
Lin, T. Y., Dollár, P., Girshick, R., He, K.,
VII. REFERENCES Hariharan, B., & Belongie, S. (2017). Feature
Pyramid Networks for Object Detection. In
Bezak, P. (2016, September). Building recognition Proceedings of the IEEE Conference on Computer
system based on deep learning. In 2016 Third Vision and Pattern Recognition (CVPR) (pp. 2117-
International Conference on Artificial Intelligence 2125).
and Pattern Recognition (AIPR) (pp. 1-5). IEEE. Redmon, J., & Farhadi, A. (2017). YOLO9000:
Jung, H., Lee, S., Park, S., Kim, B., Kim, J., Lee, I., Better, Faster, Stronger. In Proceedings of the IEEE
& Ahn, C. (2015, January). Development of deep Conference on Computer Vision and Pattern
learning-based facial expression recognition Recognition (CVPR) (pp. 7263-7271).
system. In 2015 21st Korea-Japan Joint Workshop Li, Z., Peng, C., Yu, G., Zhang, X., Deng, Y., &
on Frontiers of Computer Vision (FCV) (pp. 1-4). Sun, J. (2017). DetNet: A Backbone network for
IEEE. Object Detection. arXiv preprint arXiv:1804.06215.
Tenguria, R., Parkhedkar, S., Modak, N., Madan, Huang, J., Rathod, V., Sun, C., Zhu, M.,
R., & Tondwalkar, A. (2017, April). Design Korattikara, A., Fathi, A., & Murphy, K. (2017).
framework for general purpose object recognition Speed/accuracy trade-offs for modern
on a robotic platform. In 2017 International convolutional object detectors.
Conference on Communication and Signal
Processing (ICCSP) (pp. 2157-2160). IEEE.
Etemad, E., & GAO, Q. (2017, September). Object
localization by optimizing convolutional neural
network detection score using generic edge
features. In 2017 IEEE International Conference on
Image Processing (ICIP) (pp. 675-679). IEEE.
Mazumdar, M., Sarasvathi, V., & Kumar, A. (2017,
August). Object recognition in videos by sequential
frame extraction using convolutional neural