Real-Time Image Segmentation and Object

2024 International Conference on Advances in Computing, Communication and Applied Informatics (ACCAI) | 979-8-3503-8944-9/24/$31.00 ©2024 IEEE | DOI: 10.1109/ACCAI61061.2024.10602083

Tracking for Autonomous Vehicles

S. Priya1, Sanduru Sanath Kumar1, P. Lavanya1, Shaik Sadik1 and Ata. Kishore Kumar2
1
Student, Dept. of ECE, Mohan Babu University (Erstwhile Sree Vidyanikethan Engineering College), Tirupati, India
2
Associate Professor, Dept of ECE, School of Engineering, Mohan Babu University (Erstwhile Sree Vidyanikethan Engineering
College), Tirupati, India

E-mail : [email protected], [email protected], [email protected], [email protected],

[email protected]

Abstract- Autonomous vehicles rely on advanced perception in diverse traffic conditions, ushering in a new era of
systems to navigate dynamically changing environments. mobility that is both efficient and sustainable.
This research aims to develop a real- time image Conventional methods of object detection sometimes
segmentation and object tracking system, leveraging deep struggle to correctly recognize and classify items in real-
learning techniques, to enhance the perceptual capabilities of
autonomous vehicles. Thanks to developments in sensing,
time, especially in complex traffic settings. But the
machine learning, and artificial intelligence, autonomous advent of deep learning techniques, particularly
cars are getting closer to reality. Vehicle detection is one of convolutional neural networks (CNNs), has brought about
the most important parts of autonomous driving systems, significant advancements in computer vision and
and it's critical to the effectiveness and safety of these completely changed the way traffic-related items are
systems. The main intention of this research is to enquire identified and categorized.
how autonomous vehicles employ the cutting- edge YOLOv8
(You Only Look Once version 8) algorithm for vehicle
recognition. A real-time object detection model called
YOLOv8 has demonstrated remarkable results in
identifying and categorising items in pictures and videos. It
is the most recent model in the YOLO series and is
renowned for striking a balance between accuracy and speed
of detection.

Keywords— Python Language, Open CV, ANACONDA, Fig.1. The photos show objects before detection on the
PyCharm, Pytorch, Accuracy, R- CNN, Cutting-edge YOLO v8 left, and objects after detection on the right.
algorithm, Precision, Recall, F1-Score.
They also show bounding box coordinates, anticipated
I. INTRODUCTION confidence scores, and class names for the objects that
were discovered. There are different stages in the
This paper presents an integrated approach to vehicle development of autonomous vehicles: Level 0 has no
detection, tracking, and sign recognition for autonomous automation, whereas Level 1 has automated driving aid.
vehicles, leveraging the capabilities of YOLOv8. By As they advance to Level 2, drivers are in charge of
combining these functionalities into a unified system, we important safety duties while the car assists with
aim to enhance the autonomy and safety of self-driving acceleration and steering. At Level 3, often referred to as
vehicles, enabling them to navigate complex urban conditional automation, the car keeps an eye on its
environments, highways, and intersections with surroundings and prompts the driver to take over.
confidence and reliability. In this introduction, we provide However, further work is needed to reach Level 4, which
an overview of the challenges and motivations driving the is defined by high automation, and Level 5, which is when
development of autonomous vehicles. We then discuss the vehicles run entirely on their own without the need for
importance of accurate perception systems in enabling human intervention.
autonomous navigation and introduce the key components
of our proposed approach. Finally, we outline the II.LITERATURE SURVEY
structure of the paper, detailing the methodology,
experimental evaluations, and results, followed by a Hee Seok Lee et al. [1] developed a novel technique for
discussion of implications and future directions. Through simultaneously recognizing traffic signals and determining
the integration of YOLOv8-based vehicle detection, their exact boundaries using a convolutional neural
tracking, and sign recognition, we envision a future where network. By framing the boundary estimation problem as
autonomous vehicles can operate seamlessly and safely a 2-D posture and form class prediction task, the proposed

Authorized licensed use limited to: Bahria University. Downloaded on October 29,2024 at 16:57:32 UTC from IEEE Xplore. Restrictions apply.
solution allows a single CNN to successfully handle it. III.PROPOSED SYSTEM
The problem of object bounding box detection is
extended and made into an object pose estimation The proposed method for "Integrated Vehicle
challenge in order to achieve this goal. Convolutional Detection, Tracking, and Sign Recognition for
neural networks (CNNs) are an effective tool for handling Autonomous Vehicles Using YOLOv8" involves a
this issue by utilizing the latest developments in object systematic approach leveraging the capabilities of
detection networks. YOLOv8 for real-time object detection, coupled with
custom algorithms for vehicle tracking and sign
Using effective CNNs, Jia Li et al. [3] demonstrated in- recognition. Here's an overview of the proposed method:
field real-time traffic sign identification. Using the
structure of MobileNet and the design of faster R- Block Diagram:
convolutional neural networks (CNN), a detector was
created. Color and shape information were added into
the design to improve the localization accuracy of small
traffic lights, which are difficult for exact regression.
Lastly, an effective CNN with asymmetric kernels is
employed as the traffic sign classifier. following frames
to track the objects it has spotted. The object locations in
the three frames make up the information, which is used
by the algorithm to determine an object's trajectory.

CNN was suggested by Mohammed Ikhlayel et al. [5]

as a way to recognize traffic signs for an autonomous car
prototype's navigation. In this work, the autonomous
vehicle's navigation may be dependent on the traffic sign
recognition system. With the use of a video sensor and
deep learning techniques, an autonomous vehicle will be
able to identify and categories traffic signs according to A. Camera Feeds (Input Data):
their purpose. The autonomous vehicle will react as The system receives input data from camera feeds
suggested by the observed traffic sign and turn on the mounted on the autonomous vehicle, capturing the
actuator based on this classification. surrounding environment.
B. Vehicle Detection (YOLOv8 Algorithm):
Daniel Kold et al. explained how to modify You Only YOLOv8 is utilized for real-time detection of vehicles
Look Once (YOLO), a cutting-edge deep learning- in the camera feeds, providing bounding boxes around
based detector, to quickly and accurately recognize detected vehicles.
barcodes. The barcode detector has the ability to C. Vehicle Tracking:
identify both 1D and QR codes. With a detection rate of Detected vehicles are passed through a vehicle tracking
0.991 on the benchmark dataset of Muenster module, which employs techniques such as Kalman
BarcodeDB, the detector delivers state-of-the-art filtering and data association to maintain continuous
performance. tracks of vehicles across frames.
D. Sign Recognition (YOLOv8 Algorithm):
Sung et al. (2017) first presented the use of YOLOv8 is also employed for real-time detection and
convolutional neural network (CNN) approaches, which recognition of traffic signs and signals in the camera
were motivated by the You Only Look Once (YOLO) feeds.
algorithm. They used real fish footage in their study to E. Integrated Perception System:
assess the dependability and efficiency of the suggested The outputs from vehicle detection, tracking, and sign
techniques. The network displayed remarkable recognition are integrated into a unified perception system
performance measures, including a 93% classification deployed within the autonomous vehicle platform.
accuracy, a fish detection rate of 16.7 frames per F. Autonomous Vehicle Navigation System:
second, and an intersection over union (IoU) score of The integrated perception system provides valuable
0.634 between the projected bounding box and ground information for the autonomous vehicle's navigation
truth. Notably, this method performed better for fish system, enabling it to make intelligent decisions based on
detection than support vector machines and another the detected objects and traffic signs in the environment.
sliding window method, as well as a classifier trained on
orientationgradient feature histograms. This block diagram illustrates the flow of data and
processing steps involved in the proposed method,
highlighting the integration of YOLOv8-based vehicle
detection and sign recognition with custom algorithms

Authorized licensed use limited to: Bahria University. Downloaded on October 29,2024 at 16:57:32 UTC from IEEE Xplore. Restrictions apply.
3rd International Conference on Advances in Computing, Communication and Applied Informatics (ACCAI 2024)
Publication Partner: IEEE Chennai, Tamil Nadu, INDIA

for vehicle tracking within an autonomous vehicle

navigation system.

IV.SYSTEM ARCHITECTURE
In this method propose a comprehensive framework
utilizing deep learning techniques, specifically focusing
on the YOLOv8 model, for efficient and accurate object
detection in traffic scenes. The proposed methodology
involves a systematicpipeline consisting of dataset
collection, pre-processing, feature extraction, model Fig. 3. YOLO V8 Architecture
training using YOLOv8, and subsequent model inference.
Additionally, for video inputs, a similar process is applied V.IMPLEMENTATION
to each frame, enabling real-time object detection.
Furthermore, to enhance the analysis capabilities, the
detected objects can be passed through a CNN classifier
for further classification tasks. A complete list of all
objects detected, along with the accompanying bounding
boxes and class names, is included in the system output.
We demonstrate the effectiveness and robustness of the
suggested method in correctly identifying a variety of
items on roads, such as cars, pedestrians, and traffic signs,
by means of thorough testing and assessment.

Fig. 1. Identifies items on road for testing and assessment Fig. 4. Implementation flow diagram

 Start: This shape signifies the beginning of the

flowchart, indicating the start of the process.
 Dataset Collection: In this step, data consisting of
images or videos depicting traffic scenes are
gathered and collected.
 Pre-processing: The collected data undergoes pre-
processing, which may include tasks such as
resizing, normalization, and noise removal to
prepare the data for further processing.
Fig. 2. System Architecture  Feature Extraction: Using the YOLOv8 model,
features are extracted from the pre-processed data.
YOLO V8 Architecture YOLOv8 is specifically designed for object
Applications such as object detection, image detection tasks and efficiently extracts features
classification, and instance segmentation can make use of relevant to identifying objects in images or videos.
YOLOv8. YOLOv8 was developed by Ultralytics, who  YOLOv8 Model Training: The retrieved features are
also created the industry-defining YOLOv5 model. used to train the YOLOv8 model. This procedure
YOLOv8 features several architectural upgrades and involves using annotated data to fine-tune the
improvements over YOLOv5. It is a brand-new model's parameters in order to improve its accuracy
convolutional neural network (CNN) that has in object detection.
extremely accurate real-time object detection. This  CNN Classifier Integration: In some cases, the
method predicts bounding boxes and component detected objects may undergo further analysis or
probabilities by processing the full image using a classification using a CNN classifier. This step
single neural network, then splitting it into segments. enhances the system's ability to understand and
classify objects beyond simple detection.

Authorized licensed use limited to: Bahria University. Downloaded on October 29,2024 at 16:57:32 UTC from IEEE Xplore. Restrictions apply.
  Output: The system's final output consists of a list Furthermore, the performance of the model is evaluated
of recognized objects along with their properties, like in a variety of driving circumstances in order to
class labels and bounding boxes. This output is determine how robust itis.
useful for a variety of applications, such as traffic E. Optimization:
monitoring and autonomous driving, among others. Based on the results of the evaluation procedure,
 End: This shape indicates the end of the flow parameter changes are performed to optimize the model's
chart,marking the completion of the process. performance. In order to reduce overfitting, this entails
adjusting hyperparameters like the learning rate and the
VI.METHODOLOGY number of epochs and putting early halting and dropout
strategies into practice. Furthermore, performance and
The YOLOv8 algorithm is used in this study's compatibility of the model are confirmed through testing
methodology to detect automobiles in autonomous on multiple hardware platforms. The great degree of
vehicles. accuracy in real-time vehicle recognition and
classification of the YOLOv8 model is one of its
There are multiple steps to the process: noteworthy features. This achievement is the consequence
A. Data Collection & Pre-processing: of a careful balancing act between the model's detection
The first stage is gathering a dataset (DATASET speed and accuracy.
NAME – Cars Detection FROM KAGGLE.COM) of
pictures and videos showing different kinds of vehicles VII.METRICS OF PERFORMANCE
driven under various scenarios. To make sure they are
appropriate for input into the YOLOv8 model, the The suggested model's network performance is
photos and videos undergo pre-processing. This entails assessed using performance metrics. The performance
adjusting the pixel values to the proper sizes, measures used in this work include f1-score, recall,
downsizing the photos and videos, and enhancing the accuracy, and precision.
data to make it more substantial and diverse.
B. Model Training: Accuracy:
The pre-processed dataset is used to train the YOLOv8 Evaluates how accurately recognized signs or gestures
model. The model is specifically designed to recognize match the actual world in general. Accuracy is
and categories various car kinds; it is trained by calculated as
subjecting it to an assortment of images and videos. The
model then makes use of this dataset to produce Accuracy = Number of correct predictions
predictions for every object in the training set, such as Total Number of Predictions (1)
bounding boxes and class probabilities. The degree of
agreement between the ground truth labels and the Precision:
model's predictions is assessed, and the model's The percentage of correctly identified signs among all
parameters are changed to reduce any discrepancies that recognized signs is known as precision. Precision is
are found. This process of iteration is carried out until equal to
the model's performance on the validation set reaches a Precision= TP/(TP+FP) (2)
point where it is no longer possible to make
improvements. where FP=False Positives
C. Object tracking: TP=True Positives
This technique uses the trained model to identify and
categorize automobiles in real time. For every object in the Recall:
pictures and videos, the model predicts bounding boxes The percentage of correctly identified signs among all
and class probabilities; these predictions are then used to real signs is measured by recall.
track the cars. Multiple frames are tracked by the Recall =TP/(TP+FN) (3)
DeepSORT (Simple Online and Realtime Tracking) where FP=False Positives,
algorithm. DeepSORT tracks vehicles over time by TP=True Positives
associating detections in successive frames using the FN=False Negatives
features that the YOLOv8 model derived. The model was
trained using a real-time image in order to identify the F1-Score:
object. The F1-Score is a balanced indicator of a model's
D. Evaluation: performance that is calculated as the harmonic meanof
A range of metrics, including mean average precision precision and recall.
(mAP), recall, and precision, are used to assess the F1 − Score =2 XPrecision X Recall (4)
YOLOv8 model's performance. These metrics offer Precision+ Recall
numerical insights into the precision and efficiency of the
model in identifying and categorizing cars.

Authorized licensed use limited to: Bahria University. Downloaded on October 29,2024 at 16:57:32 UTC from IEEE Xplore. Restrictions apply.
3rd International Conference on Advances in Computing, Communication and Applied Informatics (ACCAI 2024)
Publication Partner: IEEE Chennai, Tamil Nadu, INDIA

VIII.RESULTS enhancing the perception capabilities of autonomous

vehicles. By leveraging the capabilities of YOLOv8 for
The following are the output of the simulation of object real-time object detection and integrating custom
tracking and object detection. Figures shows the result of algorithms for vehicle tracking and sign recognition, the
simulating a real time video which includes the labels: the system enables autonomous vehicles to perceive and
id (the type of vehicle) followed by confidence level, interpret their surroundings accurately and efficiently.
direction in which the vehicle is moving and finally Through comprehensive evaluations and real-world
followed by the relative direction with respect to the testing, the proposed method demonstrates its
autonomous vehicle (going far away to the vehicle or effectiveness in enhancing the autonomy and safety of
coming near to it). autonomous driving systems.

REFERENCES
[1] Zhu, X. Yan, H. Tang, Y. Chang, B. Li, and Yuan, “Moving
Object Detection With Deep CNNs,” IEEE Access, vol. 8, pp.
29729–29741, 2020, doi: 10.1109/ACCESS.2020.2972562.
[2] H. Zhu, H. Wei, B. Li, X. Yuan, and N. Kehtarnavaz, “Real-
Time Moving Object Detection in High Resolution Video
Sensing,” Sensors, vol. 20, no. 12, p. 3591, Jun. 2020, doi:
10.3390/s20123591.
[3] R. T.et al, "COVID-19 Identification and Detection from CT-
Images using AI Based Ensemble Model," (ICSES), Chennai,
India, 2022, pp. 1-10, doi: 10.1109/ICSES55317.2022.9914227.
[4] A. Balasubramaniam and S. Pasricha, “Object Detection in
Autonomous Vehicles: Status and Open Challenges”.
[5] R. T. et al "Automated Crack and Damage Identification in
Premises using Aerial Images based on Machine Learning
Techniques," (I-SMAC), Dharan, Nepal, 2022, pp. 498-504,
doi: 10.1109/I-SMAC55078.2022.9987391.
[6] S. S, C. A. S. M, J. Krishnan, R. R and V. K, "Cosmetic
Products Customisation and Customer Segmentation," 2023
International Conference on Advances in Computing,
Communication and Applied Informatics (ACCAI), Chennai,
India, 2023, pp. 1-11, doi:
10.1109/ACCAI58221.2023.10200286.
[7] Violation Detection using Tensorflow & Keras in OpenCV. In
2020 IEEE International Students Conference on Electrical,
Electronics and Computer Science (SCEECS).
[8] Chen, S., Wei, Y., Xu, Z., Sun, P., & Wen, C., 2020. Design and
Implementation of Second - generation ID Card Number
Identification Model based on TensorFlow. In IEEE
International Conference on Information Technology, Big Data
and Artificial Intelligence (ICIBA).
[9] Caveness, E., C., P. S., Peng, Z., Polyzotis, N., Roy, S., &
Zinkevich, M., 2020. TensorFlow Data Validation: Data
Analysis and Validation in Continuous ML Pipelines. In
Proceedings of the 2020 ACM SIGMOD International
Conference on Management of Data.
[10] Lu, Y., Zhang, L., & Xie, W., 2020. YOLO- compact: An
Efficient YOLO Network for Single Category Realtime Object
Detection. In 2020 Chinese Control and Decision Conference
Fig. 5. Simulation of object tracking and object detection (CCDC).
[11] Ullah, M. B., 2020. CPU Based YOLO: A Real Time Object
Detection Algorithm. In 2020 IEEE Region 10 Symposium.
IX.CONCLUSION [12] Redmon, J., Divvala, S., Girshick, R., & Farhadi, A., 2016. You
Only Look Once: Unified, Real-Time Object Detection. In 2016
IEEE Conference on Computer Vision and Pattern Recognition
The proposed method for "Integrated Vehicle Detection, (CVPR).
Tracking, and Sign Recognition for Autonomous Vehicles
Using YOLOv8" offers a robust and efficient solution for

Authorized licensed use limited to: Bahria University. Downloaded on October 29,2024 at 16:57:32 UTC from IEEE Xplore. Restrictions apply.