Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Smart System for Potholes Detection Using Computer

Vision with Transfer Learning
P. S. Ezekiel1, O. E. Taylor 2* & D. J. S. Sako3
1,2,3
Department of Computer Science, Rivers State University, Port Harcourt, Nigeria
*Corresponding Author:

Abstract:- Road potholes are extensively enormous outwardly reviewed every day and have their vehicle stream
primary disappointments out and about surface. They observed continually. Low need rustic streets are
are brought about by withdrawal and extension of the investigated not exactly once per week, and regularly have
road surface as water saturates into the ground. To undetected harm for longer time [1].
guarantee traffic wellbeing, it is critical and important to
regularly investigate and fix road potholes. This paper Road potholes are extensively enormous primary
presents a smart system for potholes detection using disappointments out and about surface. They are brought
computer vision with transfer learning. The system about by withdrawal and extension of the road surface as
starts by acquiring a pothole images as data, pre- water saturates into the ground. To guarantee traffic
processing the data by creating image annotation and wellbeing, it is critical and important to regularly investigate
image augmentation on the pothole images. We also and fix road potholes. Presently, potholes are routinely
created sub directories by creating called images and distinguished and revealed by guaranteed investigators and
annotation where we placed our training images in the primary designers. This undertaking is nonetheless, tedious
image folder and the annotated files (which was and drawn-out. Besides, the discovery results are
generated in Xml format) into the annotation folder that consistently emotional, on the grounds that they rely
we created. We then train our model using transfer altogether upon staff insight. The event of pothole has
learning. By transfer learning, we downloaded a pre- expanded quickly in uncommon climate like substantial
trained yolov3 weight file trained on a coco dataset for downpour in summer and snowfall, and affects traffic
object detection. We then set the batch_size to be equal security and street harm. It messes social up like vehicle
to 4, with a tensorflow Gpu of version ==1.13.1, number breakage and mishaps, which are causing social expenses. In
of experiments = 200, and this manner, programmed pothole discovery strategies are
train_from_pre_trained_model = pre-trained-yolov3 being read for productive pothole fix and asphalt the
(the weight file we downloaded). After training, we executives [2].
evaluated and saved the trained model. We had a
training accuracy of about 97%. We carried out a real Occupied driving, speeding or other driver mistakes
time pothole detection on a live streaming video using are primary driver of mishaps around the world; in any case,
opencv library, where we detected multiple potholes awful road conditions are additionally a critical reason. The
images. state of a street ends up being perilous because of number of
reasons like flooding, precipitation, harms caused, e.g., by
Keyword:- Potholes, Computer Vision, Transfer Learning, over-burden large vehicles, or poor actual upkeep of the
Yolov3. road. Road condition evaluation includes distinguishing and
investigating unmistakable sorts of road surface pain,
I. INTRODUCTION similar to potholes, breaks or surface changes as being
upkeep significant highlights. Full scale road highlights are
Road networks are key resources, as they support the characterized by being of traffic pertinence. For instance,
proficient progression of individuals and merchandise. hindrances are likewise traffic-significant highlights; they
However, in spite of this basic social and monetary job, additionally require identification for driver help. A pothole
numerous nations spent just half or even less of what might is an extraordinary instance of street trouble. It very well
be needed for legitimate road upkeep. Keeping up with road may be a subjectively molded primary imperfection of a
networks is frequently the duty of territorial or nearby street, and an exact ID of its "line" is commonly
government elements with spending limitations. Helpless inconceivable. They can be ambiguously laid out, yet their
support brings about street deserts, which add to monetary greatest profundity can be distinguished all the more
misfortunes, vehicle harm, and fatalities. Early recognition unequivocally. Articles like vehicles, people, cyclists,
and detailing of such imperfections would permit protection canines or felines are of explicitly characterized shapes (and
support, which would bring about lower cost, usefulness now recognized by machine learning because of appearance
benefits, and expanded wellbeing. Visual reviews are the properties) [3].
most widely recognized technique at present used to
evaluate road condition. Their viability is obliged by asset At present, the primary techniques for recognizing
accessibility and the significance of a given street. To be potholes actually depend on open announcing through
sure, high volume traffic motorways will in general be hotlines or sites, for instance, the potholes detailing site in

IJISRT21JUL1223 www.ijisrt.com 1385

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Ohio [4]. Nonetheless, this revealing typically needs exact conditions utilizing accelerometer information from
data of the dimensional and area of potholes. Additionally, implanted vehicle sensors. Their proposed publicly
this data is normally obsolete too. This paper presents a supported framework decreases the necessary organization
smart system that will detect and alert the users of potholes data transmission by deciding the grade of the road and
in Nigeria. bank point data in every vehicle to channel speed increase
parts that don't compare to pothole conditions. They assess
II. RELATED WORKS their framework on reenacted and true information, break
down tradeoffs in the quantity of vehicles and the measure
Varona et.al. (2020) proposed a deep learning of data transmission needed for precise location, and
technique to automatically recognize the various types of contrast the outcomes with the more straightforward single
road facet, and to recognize potholes from destabilizations path identification situation. The looked at result shows that
delivered by hindrances or driver activities in the group as the path keeps on expanding, the precise pace of location
detecting based application setting. Specifically, they diminishes [8].
investigate and apply distinctive deep learning models:
convolutional neural network, LSTM algorithm, and Bansal et.al. (2020) proposes a procedure DeepBus for
repository figuring models. Their experiment was conducted ongoing ID of surface anomalies on roadways utilizing
with true data, and the outcomes showed a promising Internet of Things (IoT). The proposed procedure utilizes
exactness in tackling the two problems. The accuracies are IoT sensors to identify potholes progressively. The area of
98% for convolutional neural network, and 78% for Last these potholes would be accessible on a midway facilitated
Short-Term Memory [5]. map. They looked at the exhibition of different AI models
(Logistic Regression, Support Vector Machine (SVM), K-
Anaiss et.al. (2019) proposed a virtual roadway Nearest Neighbors (KNN), Naive Bayes, Decision Tree,
network auditor (VRNI), which persistently screens Random Forest and Ensemble Voting) in view of various
roadway conditions and gives choice help to directors and boundaries (Accuracy, F-score, Precision and Recall) and
specialists. The virtual street network monitor proposes a distinguished that Random Forest is the best model for
novel roadway damage recognition strategy dependent on pothole identification with 86.8% exactness [9].
two versatile one-class support vector machine models,
which were applied on the vertical and sidelong speed Kang and Choi (2017) utilized a 2D LiDAR and
increase information. They assessed this technique on Camera in identifying potholes. To further develop the
information from a genuine arrangement on school pothole discovery accuracy, the mix of heterogeneous
transports in New South Wales, Australia. Their trial results sensor framework was utilized. They likewise utilized
show that their proposed strategy reliably distinguishes Image-based pothole recognition technique to work on the
97.5% of the road vandalization of 4% bogus caution rate precision of pothole location. The exhibition of their
that identify with considerate inconsistencies, such proposed procedure depends on blunder rate. Their test
development joints [6]. result shows that their proposed strategy have a low mistake
pace of about 0.02% [10].
Dhiman and Klette (2019) created two methods
dependent on sound system vision examination of roadway Li et.al. (2018) propose a sound system vision
conditions in front of the vehicle. They additionally planned framework for pothole identification. The framework
two models for deep learning-based pothole recognition. contains two USB cameras taking photograph at the same
They completed a trial assessment of the two proposed deep time. They utilized the acquired boundaries gotten from the
learning models as far as accuracy and review, and they camera in adjustment with checkerboard to figure the
acquired the accompanying outcomes: 96.2and 92.3 for divergence map. They projected 2-dimensional picture
accuracy and review of the main model and 65.2 and 53.2 focuses to 3-dimensional world focuses utilizing the
for the second model [3]. dissimilarity map. All focuses underneath the street surface
model was recognized as pothole district. The exploratory
Fan et.al. (2019) present a powerful pothole outcome shows that powerful identification of potholes in
recognition framework. To recognize vandalize and intact various street and light conditions [11].
roadway regions, they changed a thick difference map. To
accomplish more noteworthy divergence change Tsai and Chatterjee (2018) presents a pothole
productivity, they used brilliant area search and dynamic recognition strategy utilizing 3D asphalt information and a
programming to gauge the change boundaries. For the watershed technique. They complete tests utilizing the 3D
extraction of potential intact roadway regions from the information gathered on tenth Street, Atlanta, Georgia and 6
changed divergence map, they received Otsu's thresholding mi of street on U.S. 80, Savannah, Georgia, has shown a
strategy. Their test results show that the correct recognition 94.97% exactness, 90.80% accuracy, and 98.75% review. It
accuracy of their proposed framework is around 98.7% and has been exhibited that the proposed strategy is promising
the general pixel-level precision is roughly 99.6% [7]. for pothole location and can give a dependable technique to
pothole identification, particularly when 3D asphalt
Fox et.al. (2017) create a publicly supported information have been gathered for break discovery and as
framework to distinguish and confine potholes in multi-path of now accessible [12].

IJISRT21JUL1223 www.ijisrt.com 1386

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
III. METHODOLOGY

Labelme Image Transfer Learning

Pothole Images Annotation Tool with Yolo-3

Image
Data Acquisition Image Annotation Model Training
Augmentation

Real Time Detection Model Evaluation

OpenCV
Figure 1 Architecture of the Proposed System

Data Acquisition: The dataset consist of 1200 images of translation, affine transformation zooming and sharing of
road with potholes. These images was collected online using images. This was used as a result of generating more images
google search engine to search for potholes images in Port and changing the position of the images so as to obtain a
Harcourt City, Rivers State Nigeria. The acquired dataset better recognition of our trained model.
will be categorized into two which is a training data and a
𝑥 𝑦
testing data. 𝑥 = 𝑓𝑥 . 𝑧 + 𝑥𝑐 , 𝑦 = 𝑓𝑦 . 𝑧 + 𝑦𝑐 , … Image Translation

Image Annotation: Image annotation is the task of Where x and y are the image pixel, fx and fy are the focal
annotating an image with labels. The labels are used to give lengths in x and y coordinate direction, and (xc, yc) is the
information to a computer vision model about the given principal point in the image plane.
images that is to be used as an input data. For annotation of
our training and testing images, we will be using Labelme Model Training: The model will be trained using a pre-
annotation tool in annotating our images. The labelme will trained yolo-3 coco weight model that was trained on a coco
be used in segmenting our images (training and testing), dataset for different object detection. We will be optimizing
drawing bounding boxes and generating coordinates for the parameters of the pre-trained model weights to be a
each of the training image. batch_size of 4, 200 number of experiments, and the number
of object names to be one (1). This process of training a
Image Augmentation: image augmentation is used in new model from a pre-trained model weights is called
expanding our dataset. We also used image augmentation transfer learning.
parameters for rotation, scaling, cropping, flipping,

IJISRT21JUL1223 www.ijisrt.com 1387

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Figure 2 Architecture of Yolo v3

Figure 3: Pseudo Code of Yolo v3 Algorithm

Model Evaluation: This involves the testing and evaluation

the general performance of the model based on accuracy, Where TP is True Positive Result, TN is the True Negative
precision, cross-validation. We will evaluate the Result; FP is the False Positive Result, and FN is the False
performance of our trained model based on accuracy, and Negative Result.
cross validation.
Real Time Detection: We will carry out a real time
𝑇𝑃 detection of potholes on Port Harcourt City in Rivers State
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃 + 𝑇𝑃 using Open CV in loading a live video from a web cam, we
𝑇𝑃 + 𝑇𝑁 will show the web cam to major roads in Port Harcourt in
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁 detecting multiple potholes as possible.

IJISRT21JUL1223 www.ijisrt.com 1388

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
IV. RESULT AND DISCUSSION folder. The images folder contains the pothole images while
the annotation folder contains an xml file for each of the
This system proposes a real time system for potholes images that we are to use for both training and testing of our
detection using computer vision. The system starts by model. The xml file in the annotation folder contains the
acquiring images of potholes in Port Harcourt City Rivers coordinate (x, y) of each of the images for both training and
State. The dataset comprises of 1200 images of potholes. testing data. This coordinates is to enable us to draw a
These images was pre-processed using image annotation bounding box of various potholes images. After these
tool and image augmentation. The image annotation tool processes, we then train our model using transfer learning.
used in this work is that of the labelme annotation tool. We By transfer learning, we downloaded a pre-trained yolov3
carried out the annotation of each of images we acquired. weight file trained on a coco dataset for object detection. We
We performed segmentation of the images we will be using then set the batch_size to be equal to 4, with a tensorflow
as data by drawing a bounding box of each of the images. Gpu of version ==1.13.1, number of experiments = 200, and
We also label each of the images, so as to make it known to train_from_pre_trained_model = pre-trained-yolov3 (the
our proposed model for easy identification of potholes. We weight file we downloaded). Figure 4 shows the loss values
also perform image augmentation so as to rotate, zoom, obtained at the first seven training steps. Figure 5 shows the
translate, and transform the training images so as to make it evaluation of our model performance. Figure 7 shows the
easy for the model to detect potholes despite the position of matrix evaluation of our trained model while figure 8, and 9
the image. After the processes we then divided our dataset shows real time detection of potholes images that is carried
into training set and testing set. We allocate 80% of the out using opencv, and figure 10 shows the models
images to our training folder and 20% of the remaining data evaluation in terms of accuracy, where model2 has the
to our test folder. We also created two sub folders called highest accuracy of about 97%.
images and annotation for both the training and testing

Figure 4: Training processes with loss values

IJISRT21JUL1223 www.ijisrt.com 1389

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Figure 6: Evaluation of trained model

IJISRT21JUL1223 www.ijisrt.com 1390

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Figure 7: Matrix evaluation in terms of precision

Figure 8: Detected images of Potholes

IJISRT21JUL1223 www.ijisrt.com 1391

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Our trained model detected more than one pothole in images with confidence level of 66 %and 88%.

Figure 9: Detected Images of Potholes

Here, our model shows 97% confidence level while detecting potholes.

Figure 10: Bar chat Representation of Training Model

This shows the model performance in terms of accuracy the annotation folder that we created. We then train our
during training. We 6 models just to find the model with the model using transfer learning. By transfer learning, we
best accuracy result. The bar chat above shows that the downloaded a pre-trained yolov3 weight file trained on a
model2 has the highest training accuracy result ahead other coco dataset for object detection. We then set the batch_size
models. to be equal to 4, with a tensorflow Gpu of version ==1.13.1,
number of experiments = 200, and
V. CONCLUSION AND FUTURE WORK train_from_pre_trained_model = pre-trained-yolov3 (the
weight file we downloaded). After training, we evaluated
This paper presents a smart system for potholes and saved the trained model. We carried out a real time
detection using computer vision with transfer learning. The pothole detection on a live streaming video using opencv
system starts by acquiring a pothole images as data, pre- library, where we detected multiple potholes images. This
processing the data by creating image annotation and image work can further be extended by creating an android app
augmentation on the pothole images. We also created sub that will detect potholes and use google map in saving the
directories by creating called images and annotation where location of the detected potholes to database.
we placed our training images in the image folder and the
annotated files (which was generated in Xml format) into

IJISRT21JUL1223 www.ijisrt.com 1392

Volume 6, Issue 7, July – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
REFRENCES

[1]. J. Eriksson, L. Girod, B. Hull, R. Newton, S. Madden,

H. Balakrishnan , “The pothole patrol: using a mobile
sensor network for road surface monitoring”, In:
proceedings of the 6th International Conference on
Mobile systems, applications, and services, ACM, 29–
39, 2008
[2]. C. Koch, K. Georgieva, V. Kasireddy, B. Akinci, and
P. Fieguth, “A review on computer vision based defect
detection and condition assessment of concrete and
asphalt civil infrastructure,” Advanced Engineering
Informatics, vol. 29, no. 2, pp. 196–210, 2015.
[3]. A. Dhiman, R. Klette, “Pothole Detection Using
Computer Vision and Learning”, Ieee Transactions On
Intelligent Transportation Systems, 21(8), 3536-3550,
2019.
[4]. R. Madli, S. Hebbar, P. Pattar, and V. Golla,
“Automatic Detection and Notification of Potholes and
Humps on Roads to Aid Drivers,” IEEE Sensors
Journal, vol. 15, no. 8, pp. 4313–4318, 2015.
[5]. B. Varona, A. Monteserin, A. Teyseyre “A deep
learning approach to automatic road surface
monitoring and pothole detection”, Personal and
Ubiquitous Computing, 24(4), 519-534, 2020.
[6]. A. Anaissi, N. L. Dang Khoa, T. Rakotoarivelo., ·
M. M. Alamdari, · Y. Wang, “Smart pothole detection
system using vehicle-mounted sensors and machine
learning”, Journal of Civil Structural Health
Monitoring, 9(1), 91-102, 2019.
[7]. R. Fan, U. Ozgunalp, B. Hosking, M. Liu, I. Pitas,
“Pothole Detection Based on Disparity Transformation
and Road Surface Modeling”, Ieee Transactions On
Image Processing, vol. 29, 897-908, 2019.
[8]. A. Fox, B.V.K. Vijaya Kumar, J. Chen, F. Bai, “Multi-
lane Pothole Detection from Crowdsourced
Undersampled Vehicle Sensor Data”, IEEE
Transactions on Mobile Computing 16(12), 3417-
3430, 2017.
[9]. K. Bansal, K. Mittal, G. Ahuja, A. Singh, S. S. Gill,
“DeepBus: Machine learning based real time pothole
detection system for smart transportation using IoT”,
Internet Technology Letters 3 (3), vol.156, 1-6, 2020
[10]. B. Kang, S. Choi, “Pothole Detection System Using
2D LiDAR and Camera”, 2017 Ninth International
Conference on Ubiquitous and Future Networks
(ICUFN), 744-746, 2017.
[11]. Y. Li, C. Papachristous, D. Weyer, “Road Pothole
Detection System Based on Stereo Vision”, Naecon
2018-IEEE National Aerospace and Electronics
Conference, 292-297, 2018.
[12]. Y. Tsai, A. Chatterjee, “Pothole Detection and
Classification Using 3D Technology and Watershed
Method”, Jouranl of Computing in Civil. Engineering,
2018, 32(2), 04017078, 2018

IJISRT21JUL1223 www.ijisrt.com 1393