Proceedings of the Fifth International Conference on Intelligent Computing and Control Systems (ICICCS 2021)

IEEE Xplore Part Number: CFP21K74-ART; ISBN: 978-0-7381-1327-2

Pothole and Object Detection for an Autonomous

Vehicle Using YOLO
Kavitha R Nivetha S
Associate Professor, Department of Electrical and PG Scholar, Department of Electrical and Electronics
Electronics Engineering Engineering
Kumaraguru College of Technology,Coimbatore, India Kumaraguru College of Technology,Coimbatore, India
[email protected] [email protected]
2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS) | 978-1-6654-1272-8/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICICCS51141.2021.9432186

learning techniques implemented on the area of computer

vision which has grown immensely in the field of: Video
surveillance, Industrial automation, Self-driving vehicle,
Abstract— Object Detection is an key software military, medical industry etc these development
and a fundamental task for an autonomous accomplished excellent results. Using this process object can
driving system that provides remarkable change be localized, predicted and classified based on the object that
in computer vision. İn recent years, company’s are is detected [1]. Object detection mainly classified based on
planning to launch autonomous vehicle in an full traditional detector and deep learning methods[2] using region
swing that’s the most important ascpets for object of interest, Traditional proposal that includes : Viola Jones,
detection and one of most challenging task for Histogram of Gradient, Scale Invariant Feature Transform,
locating specific object from from multiple objects Haar Feature, Speeded up Robust Feature, Shape Context,
in a specific scenario. The computer vision and
Local Binary Feature, and Covariance. Also includes complex
machine learning algorithm is the important tool
Deep Learning Detector that includes: Regional Based CNN
for detecting objects in and around the
(RCNN) [3][4] which is too slow in real time prediction,
environment.In this paper, which consists of two
Spatial Pyramid Pooling Network, Fast Regional Based
parts The first part is implemented on object
RCNN (Fast RCNN), Faster Region Based CNN (Faster
detection in the surrounding with Yolo (You Only
Look Once)Algorithm provides exact classification
RCNN) [5], You Only Look Onces (YOLO) where image
and position which is configured on newly created processing for object detection is faster, Single Shot Multi
datasets for classes of object: a car, a person, a Box Detector (SSD), Retina Net, Squeezedet. Implemented in
truck, a bus, traffic light, motorcycle, pothole, python, C++, Matlab environment that extracts features of
wetland uses the Convolutional Neural Network image and video application, Background Modelling, and
and max-polling layer for prediction that segmentation also involved in image processing concepts
improves detecting of small target and these deep provides good accuracy for feature extraction and
learning technique provides a high accuracy for classification of detected image [6]. That requires large
detecting real world. Detecting potholes in Indian amount of pre-trained image dataset for both testing and
road help the autonomous vehicle to move training the model this help to solve complex vision-related
smoothly without getting struck in the potholes.In task that has grabbed attention by many researchers pay a way
part two of the proposed method is implemented for detecting single/multiple images in the video frame for
on Raspberry pi4 a popular embedded computer better interpretation as a whole. The dataset include ImageNet,
board explores suitability for the running objects . COCO, and VOC make the autonomous vehicle to without
That solves the real world problems and improves human error. For object detection vehicle uses sensor like:
the impact on detecting objects. Knowing pothole radar, lidar, camera with better calibration
and wetland detection for self-driving vehicle is In this paper, we used YOLOv3 algorithm to train our
needed badly to solve the road lay problems like: CNN model to make them to adapt to the complex
accident, slowing down the transport system these environment in the real world and was experimented with the
are solved by deep learning. darknet53 of custom weights. Finally, it improves the
Keywords—Autonomous Vehicle; Deep Learning Neural Network; detection of models with trained dataset of: motorcycle,
CNN; YOLO Algorithm; Object Detection; Image Processing people, pothole, car, traffic light, truck, bus, and wetland
analyzed bases on neural network provide great accuracy and
I. INTRODUCTION establishment. Extracting each feature from the image is
scanned with probability classes that hints colour, intensity,
Today artificial intelligent used in many real time texture, and density of the image [7] detected region is
application and there are numerous developments in deep identified which becomes highly accurate computer vision.

978-0-7381-1327-2/21/$31.00 ©2021 IEEE 1585

Authorized licensed use limited to: National University of Singapore. Downloaded on July 04,2021 at 08:14:27 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fifth International Conference on Intelligent Computing and Control Systems (ICICCS 2021)
IEEE Xplore Part Number: CFP21K74-ART; ISBN: 978-0-7381-1327-2

Hence, it is also important for detecting poor road condition

like pothole [8], crackpot and wetland because self-driving
vehicle tend to cause accident in rainy or in bad weather
condition. And it’s necessary for self-driving vehicle to get
appropriate information while travelling in order to go to the
detection.
This system is developed using Raspberry pi 4B is the
power development board with high performance ARM cortex
A72 4x 1.5GHZ quad core processor that supports micro SD
card and also support mouse, keyboard and display interface
for use its similar to the LINUX operating system because
object detection in real world requires lot of processing speed
and frame rate are useful in practical application. The paper is
organized in follows as Section2 explains method of
implementation, Section3 describes the software solution,
Section4 presents the hardware results, In final section
summarize the final results and gives direction to the final
work.

II. PROPOSED METHODOLOGY

Fig.1. Flow Chat of Object Detection Module
The method used for training and testing model is by
YOLO Algorithm.The YOLOv3 algorithm is trained on
creating labelled training set which detectes around 8 objects
with classes : car, bus, truck, person, pothole, wetland, From the flow chat detection of the desired object based on
motorcycle, trafficlight. Training images were labelled and the YOLOv3 algorithm is needed some modification to the
images are collected using PASCAL VOC [9]image database YOLO network by creating the training set and need to
with xml format which gives the annotation based width, validation the new model. From flow chat, we needed to open
height, dept and size of image with boundary box in form SxS labelling tool [10] available in python helps to build the
grid. Camera capture the object as the input image where, boundary box (x, y, w, h, c) which represents height, width,
video process in frame by frames with determined object centre co-ordinates and confident score with the predicted
created by capturing the traffic in coimbatore city after classes. After creating and labelling the image form a zip
collecting the image each boundary box is marked with folder there is need to make some changes in the YOLO
LabelImg labelled tool [10] of interst in each image configuration file based on desired classes of the object with
respectively. Captured frame from the scene, predicted object filters in images. Based on the dataset the training is done with
are resized then its forwarded to the YOLO node through the iteration and sub division parameter. For, filtration of training
neural network with more efficient Darknet 53 convolutional the epochs the class parameter must be determined with the
layer for object detection.The detected object is assised with predicted class objects.
class co-ordinates and its boundary box. Each image from a
SxS grid with boundary box .Box with higher confidence of Filters = (classes + 5) * no. of classes (1)
IOU grids score as output.In the Fig. 1 the flow chat for object
detection is shown. For, training of the weights darknet53.conv.74 [9] should
be downloaded for pre-training the weights which is available
in the internet and minimum number of epochs is determined
by the calculation. And there must be minimum no .of 16000
epochs for training the data set in YOLO. The training lasted
for 24hours.

No. of .epochs = 2000* classes (2)

After training the neural network captured image and

videos frames are uploaded in the training folder. By
estimating such algorithm object detection is done with
framework which can solves the real world problems and
improves the impact on detecting objects. Knowing pothole

978-0-7381-1327-2/21/$31.00 ©2021 IEEE 1586

Authorized licensed use limited to: National University of Singapore. Downloaded on July 04,2021 at 08:14:27 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fifth International Conference on Intelligent Computing and Control Systems (ICICCS 2021)
IEEE Xplore Part Number: CFP21K74-ART; ISBN: 978-0-7381-1327-2

and wetland detection for self-driving vehicle is needed badly

to solve the road lay problems.

III. SOFTWARE IMPLEMENTATION

The software solution is based on darknet on python

programming language and desktop computer is used for
training with estimated object detection algorithm in a self-
driving vehicle [11]. The frames captured from camera is
forwarded to YOLO algorithm and output is identified based
on number of detected objects in the frame.

Fig.2.(d). Wethole Detection with Custom Dataset

The proposed algorithm can detect all of them with high

score and detect the boundary box accurately. The Yolo v3
algorithm was trained and evaluated by NVidia GeForce GTX
GPU which provide training and validation images weights
were initialized using pre-trained dataset[12]. Its trained on
epochs for training that provides input video stream from the
dashboard camera display that confirmed yolo that can be
implemeded in the real time system.

Fig.2.(a) Visual Perception of the Object Detected

Fig.3.(a). Training the Epochs on Training Dataset

Fig.2.(b) Visual Perdection of the Object Detected

Fig.3.(b). Training the Epochs on Validation Dataset

IV. HARDWARE IMPLENTATION

In the proposed system first we need to import raspberry
operating system image to the SD card [13]which as thehard-
Fig.2.(c) Pothole Detection with Custom Dataset drive process is continued by enabling SSH.Vnc viewer is

978-0-7381-1327-2/21/$31.00 ©2021 IEEE 1587

Authorized licensed use limited to: National University of Singapore. Downloaded on July 04,2021 at 08:14:27 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fifth International Conference on Intelligent Computing and Control Systems (ICICCS 2021)
IEEE Xplore Part Number: CFP21K74-ART; ISBN: 978-0-7381-1327-2

downloaded to the local disk. Then installed all required

python libaries by upgrading , tensorflow is cloned to the local
disk with pre-trained weight that was trained based on epochs
calacated dataset are installed by Raspberry pi command
prompt and configuration is changed based on the required
classes in yolo config file. Then python detection demo py file
is processed for multiple object including car, person,
motorbike, truck, pothole, wetland, bus, trafficlight, bus
captured by the camera. which is installed in Raspberry pi 4B
with Yolo object detection algorithm it detects the object in
the real world. The single board Raspberry pi circuit make it
easier to place the camera sensor with the re-placement to
computer make images more predicted. The board is presized
in form of small credit card. Object detection is done with web
application and output is seen on the display on 7 inch display
board interfaced with raspberry pi board.

Fig.6. Exprimental Result 1 on Raspberrypi Interfaced with 7 inch

Display

Fig.4. Raspberry pi 4B

Fig.5. 7 Inch Display Board of Raspberry pi Fig.7. Experimental Result 2 on Raspberry pi Interfaced with 7 inch
Display based on video captured

978-0-7381-1327-2/21/$31.00 ©2021 IEEE 1588

Authorized licensed use limited to: National University of Singapore. Downloaded on July 04,2021 at 08:14:27 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fifth International Conference on Intelligent Computing and Control Systems (ICICCS 2021)
IEEE Xplore Part Number: CFP21K74-ART; ISBN: 978-0-7381-1327-2

V. CONCULSION [10] Mr. Akash S. Kadam1, Miss. Anjali H. Ghogare “Pothole Detection and
Notification Using Smartphone for Self-driving Vehicles,Drivers and
In this paper, method of object detection in self-driving Road Authorities” 2018 International Journal of Advanced Research in
vehicle which is classified into two parts one for detecting the Computer and Communication Engineering
objects using YOLO algorithm and its configured to detect the [11] Shaobin Chen ,Wei Lin1“Embedded System Real-Time Vehicle
Detectionbased on Improved YOLO Network” 2019 IEEE
object classes like: car, person, truck, bus, pothole, wetland,
[12] Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental
trafficlight, motorcycle with increased computation power improvement. arXiv preprint arXiv:1804.02767
with more resourses. Pre-trained images with help tensorflow [13] Sumeet Sanjay Walam, Siddhi Prakash Teli, Mr.Suhas M Patil “Object
of labelled PASCAL VOC image database. And second Detection and seperation Using Raspberry PI” 2019 IEEE
method is implemeded on Raspeberry pi 4 a popular
embedded computer board explores suitability for the running
objects and displayed on 7inch display board of raspberrypi.
This solves many real world problems and improves the
impact on detecting objects. Knowing pothole and wetland
detection for self-driving vehicle is needed badly to solve the
road lay problems like: accidient, slowing down the transport
system these are solved by deep learning. This provide more
benifits to the ADAS system to drive the vehicle in more saver
ways with solid base of object detection module in self-
driving cars.
İn future more objects like building , ambulance, auto-
rickshaw, rocks in the road, heavy load plastic bag on roadway
can be trained for object detection for making the autonomous
driving system more efficient and travelling on a saver ways.

References.
[1] Chinthakindi Balaram Murthy, Mohammad Farukh Hashmi , Neeraj
Dhanraj Bokde and Zong Woo Geem “Investigations of Object
Detection in Images/Videos Using Various Deep Learning Techniques
and Embedded Platforms—A Comprehensive Review” 2020
www.mdpi.com/journal/applsci
[2] Ceren Gulra Melek, Elena Battini Sonmez, Songul Albayrak” Object
Detection in Shelf Images with YOLO” 2019 IEEE
[3] Huansheng Song, Haoxiang Liang, Huaiyu Li, Zhe Dai and Xu Yun
“Vision-based vehicle detection and counting system using deep
learning in highway scenes” 2019 spinger Song European Transport
Research Review https://doi.org/10.1186/s12544-019-0390-4
[4] Rosa Andrie Asmara, Bimo Syahputro, Dodit Supriyanto , Anik Nur
Handayani, “Prediction of Traffic Density Using YOLO Object
Detection and Implemented in Raspberry Pi 3b + and Intel NCS 2” 978-
1-7281-8131-8/20/$31.00 ©2020 IEEE
[5] Chengji Liu1 ,Yufan Tao1 ,Jiawei Liang1 ,Kai Li1 ,Yihang Chen
“Object Detection Based on YOLO Network” 2018 IEEE 4th
Information Technology and Mechatronics Engineering Conference.
[6] Mario Gluhaković, Marijan Herceg “Vehicle Detection in the
Autonomous Vehicle Environment for Potential Collision Warning”
978- 1-7281 -82 59-9/2 0/$31.00 ©2020 IEEE Authorized licensed use
limited to: University of New S outh Wales. Downloaded on August
09,2020
[7] B. Hardjono, A. E. Widjaja, R. Kondorura, and A. M. Halim,
“DeepLearning Methods,” 2018 IEEE 9th Annu. Inf. Technol.
Electron.Mob. Commun. Conf., pp. 556–562, 2018.
[8] L. Lou, Q. Zhang, C. Liu, M. Sheng, Y. Zheng, and X. Liu,“Vehicles
detection of traffic flow video using deep learning,” Proc.2019 IEEE 8th
Data Driven Control Learn. Syst. Conf. DDCLS 2019, pp. 1012–1017,
2019, doi: 10.1109/DDCLS.2019.8908873
[9] Chandan G, Ayush Jain, Harsh Jain, Mohana “Real Time Object
Detection and Tracking Using DeepLearning and OpenCV” 2018 IEEE
Xplore Proceedings of the International Conference on Inventive
Research in Computing Applications

978-0-7381-1327-2/21/$31.00 ©2021 IEEE 1589

Authorized licensed use limited to: National University of Singapore. Downloaded on July 04,2021 at 08:14:27 UTC from IEEE Xplore. Restrictions apply.