REAL-TIME POTHOLE DETECTION SYSTEM

USING IMPROVED CNN MODEL – SSD

A
Project Report
Submitted in partial fulfilment of the
Requirements for the award of the Degree of

BACHELOR OF ENGINEERING
IN
INFORMATION TECHNOLOGY
By

KHATEEB MOHAMMED ZAHID PARVEZ - 1602-20-737-076

BATHINI AJAY GOUD - 1602-20-737-310

Under the guidance of

Ms. B. Leelavathy

Assistant Professor

Department of Information Technology

Vasavi College of Engineering (Autonomous)
ACCREDITED BY NAAC WITH 'A++' GRADE
(Affiliated to Osmania University)
Ibrahimbagh, Hyderabad-31
2024
 Vasavi College of Engineering (Autonomous)
ACCREDITED BY NAAC WITH 'A++' GRADE
(Affiliated to Osmania University)
Hyderabad-500 031
Department of Information Technology

DECLARATION BY THE CANDIDATE

We, Khateeb Mohammed Zahid Parvez and Bathini Ajay

Goud bearing hall ticket number, 1602-20-737-076 and 1602-20-737-
310 hereby declare that the project report entitled Real-time project
pothole detection system on vehicle using improved CNN model-
SSD under the guidance of Ms. B. Leelavathy, Assistant Professor
Department of Information Technology, Vasavi College of Engineering,
Hyderabad, is submitted in partial fulfilment of the requirement for the
award of the degree of Bachelor of Engineering in Information
Technology

This is a record of bonafide work carried out by us and the

results embodied in this project report have not been submitted to any
other university or institute for the award of any other degree or
diploma.

Khateeb Mohammed Zahid Parvez

1602-20-737-076

Bathini Ajay Goud

1602-20-737-310
 Vasavi College of Engineering (Autonomous)
ACCREDITED BY NAAC WITH 'A++' GRADE
(Affiliated to Osmania University)
Hyderabad-500 031
Department of Information Technology

BONAFIDE CERTIFICATE

This is to certify that the project entitled Real-time pothole detection

system using improved CNN models being submitted by Khateeb
Mohammed Zahid Parvez and Bathini Ajay Goud bearing 1602-20-
737-076 and 1602-20-737-310 in partial fulfilment of the requirements
for the award of the degree of Bachelor of Engineering in Information
Technology is a record of bonafide work carried out by them under my
guidance.

Ms. B. Leelavathy Dr. K. Ram Mohan Rao,

Assistant Professor Professor,
Internal Guide HOD, IT
 ACKNOWLEDGEMENT
The satisfaction that accompanies that the successful completion
of the project would not have been possible without the kind support
and help of many individuals. We would like to extend our sincere
thanks to all of them.

It is with immense pleasure that we would like to take the

opportunity to express our humble gratitude to Ms. B. Leelavathy,
Assistant Professor, Information under whom we executed this
project. We are also grateful to Dr. S. Sree Lakshmi, Assistant Professor
(Sr. Scale), Information Technology for her guidance. Their constant
guidance and willingness to share their vast knowledge made us
understand this project and its manifestations in great depths and helped
us to complete the assigned tasks.

We are very much thankful to Dr. K. Ram Mohan Rao, Pr,

Information Technology, for his kind support and for providing
necessary facilities to carry out the work.

We wish to convey our special thanks to Dr. S. V. Ramana,

Principal of Vasavi College of Engineering for giving the required
information in doing my project work. Not to forget, we thank all other
faculty and non-teaching staff, and my friends who had directly or
indirectly helped and supported me in completing my project in time.

We also express our sincere thanks to the Management for

providing excellent facilities. Finally, we wish to convey our gratitude
to our family who fostered all the requirements and facilities we need.
 Abstract

In the dynamic landscape of computer vision, object detection serves as a

cornerstone task empowering machines to identify and localize multiple objects
within images. This project embarks on the development of an object detection
system utilizing the Single Shot Detector (SSD) framework, celebrated for its
efficiency, speed, and accuracy.

SSD operates by generating fixed sets of default bounding boxes (anchors)

across multiple feature maps and directly predicts class scores and bounding box
offsets for each default box. This unique architecture enables SSD to efficiently
capture objects at various scales within a single forward pass, distinguishing it from
two-step detectors relying on region proposal networks.

The project's primary aim is to harness SSD's capabilities to detect and

classify road damages, particularly emphasizing scenarios necessitating precision
and real-time processing. To accomplish this, the project integrates a pre-trained
ResNet34 backbone, renowned for its depth and performance in image recognition
tasks.

Furthermore, the system extends its functionality by seamlessly integrating

the ESP32 camera module and the Neo6M GPS Module, enabling real-time video
capture and precise geo-location tracking. Additionally, integration with Google
Maps facilitates dynamic assessment of detected objects' spatial context, enhancing
situational awareness and decision-making.

Beyond mere detection, the system furnishes insights into object bounding
box coordinates and associated labels, fostering a comprehensive understanding of
detected entities' spatial context and semantic relevance within the image.

By harnessing the power of modern deep learning frameworks such as

PyTorch and leveraging meticulously curated datasets, this project endeavors to
push the boundaries of object detection, unlocking applications across diverse
domains ranging from autonomous driving to surveillance and beyond.
List of Figures vii
List of Tables viii
List Of Abbreviations xi
1 INTRODUCTION 1
1.1 Problem Statement – Overview 1
1.2 Proposed Work 1
1.3 Scope & Objectives of the Proposed Work 2
1.4 Organization of the Report 3
2 LITERATURE SURVEY 5
3 PROPOSED WORK 7
3.1 Block Diagram 7
3.2 Algorithm 9
4 EXPERIMENTAL STUDY 14
4.1 Data Sets 14
4.2 Software Requirements 15
4.3 Hardware Requirements 16
4.4 Preprocessing 18
4.5 Parameter Setting 21
4.6 Results 24
4.7 Analysis 27
5 CONCLUSION AND FUTURE SCOPE 32
REFERENCES 34
 LIST OF FIGURES

Fig. No Description Page No

i Block diagram of proposed work 7

ii Use Case Diagram 8

iii Dataset after applying Augmentation Techniques 15

iv Neo6M GPS Module interfacing with ESP32 16

v Neo6M GPS Module 17

vi ESP32 Camera Module 17

vii SSD Model Architecture 22

viii Detection of Potholes in Different Lighting Conditions by our 24

SSD model
xi Detection of Potholes of Different Sizes, Shapes, and 25
Dimensions
x Detection of False Objects like Trees, Shadows, Vehicles, 25
Healthy Roads
xi Comprehensive website integrated with Google Maps API for 26
analyzing results and viewing potholes in a route

xii Iterations vs Train Loss and Epoch vs mAP@50 28

xiii Iterations vs Train Loss and Epoch vs mAP@50 29

xiv Iterations vs Train Loss and Epoch vs mAP@50 29

xv Iterations vs Train Loss and Epoch vs mAP@50 30

xvi Iterations vs Train Loss and Epoch vs mAP@50 30

xvii Iterations vs Train Loss and Epoch vs mAP@50 30

xviii Iterations vs Train Loss and Epoch vs mAP@50 31

 LIST OF TABLES

Table No. Table Name Page No

I Results obtained while training the SSD model 27
 LIST OF ABBREVIATIONS

1. SSD – Single Shot Detector

2. YOLO – You Only Look Once
3. SGD - Stochastic Gradient Descent
4. ADAM – Adaptive Movement Estimation
5. CNN – Convolution Neural Networks
6. mAP – Mean Average Precision
 1. INTRODUCTION

1.1 Problem Statement - Overview

The problem statement for the real-time pothole Detection System addresses
the pressing issues surrounding manual pothole detection and repair processes.
Potholes pose significant safety hazards on roads, leading to accidents, injuries, and
vehicle damage. Moreover, they contribute to the deterioration of infrastructure,
necessitating frequent and costly repairs. Current methods of pothole detection and
repair are often inefficient and reactive, relying on manual inspections or citizen
reports, which may result in delays and inadequate responses to road hazards. These
challenges underscore the critical need for an innovative solution that can automate
pothole detection processes, enhance road safety, and optimize infrastructure
maintenance efforts.
In response to these challenges, the proposed project seeks to develop a
realtime pothole detection system leveraging advancements in computer vision and
machine learning technologies. By automating the detection process, the system
aims to improve the efficiency and effectiveness of pothole identification and repair
initiatives. This proactive approach not only enhances road safety by minimizing
the risk of accidents but also optimizes resource allocation and reduces maintenance
costs associated with road infrastructure. The proposed solution holds promise for
revolutionizing pothole management practices, offering a scalable and sustainable
solution to address the longstanding challenges faced in the domain of
transportation infrastructure maintenance.

1.2 Proposed Work

The proposed solution for pothole detection relies on the Single Shot
Detector (SSD) model within the PyTorch framework. PyTorch is chosen for its
flexibility, ease of use, and strong community support, facilitating seamless
integration with SSD for both training and deployment. As for the architecture,
ResNet34 variants are employed as the backbone due to their effectiveness in
feature extraction, essential for accurately detecting potholes amidst diverse image
features. SSD is selected as the primary model because of its capability to predict
bounding boxes at different scales in a single pass through the network, making it
suitable for real-time applications like pothole detection. Additionally,

1
experimenting with different momentum values is conducted to optimize the
model's training process, ensuring better performance. In summary, the solution
combines the strengths of PyTorch, ResNet34 architecture, and SSD model to
propose a robust system for pothole detection.
This innovative system not only alerts drivers to the presence of potholes
but also provides a mechanism to instantly notify relevant officials about damaged
roads. This proactive approach to road damage monitoring and reporting aims to
enhance overall road safety and expedite the necessary repairs, ultimately
contributing to more reliable and secure road infrastructure.

1.3 Scope & Objectives of the Proposed Work

The proposed work aims to address the challenge of real-time pothole
detection using advanced CNN models, with a defined scope and set of objectives.
The scope of the project encompasses various stages, starting with the collection
and preprocessing of a diverse data set containing images of potholes. This dataset
will be crucial for training and evaluating the performance of the developed
models.
In terms of model development, the project seeks to explore different
frameworks, architectures, and models suitable for pothole detection. Specifically,
the focus is on selecting and finetuning CNN models to achieve optimal
performance in detecting potholes across different environmental conditions and
road surfaces. This includes experimenting with various convolutional neural
network architectures to find the most effective approach.
Once the models are trained on the collected dataset, the next phase involves
evaluating their performance using predefined metrics such as [email protected]. The
objective here is to ensure that the models can accurately detect potholes while
minimizing false positives and negatives. This evaluation phase will help in
identifying any shortcomings and areas for improvement.
Following the initial training and evaluation, the project will implement post
tuning strategies to further enhance the performance of the trained models. This
may involve finetuning hyperparameters, implementing postprocessing techniques,
or exploring other optimization methods to improve the accuracy and robustness of
the system.

2
 One of the key objectives of the proposed work is to achieve real-time
performance, enabling the system to process live camera feeds and detect potholes
on the fly. This requires developing efficient algorithms and optimizing the
computational resources to minimize processing time and latency.
Additionally, the project aims to integrate the pothole detection system with
Google Maps, to provide users with enhanced functionalities or visualizations. This
integration will enhance the usability and usefulness of the system, making it more
valuable for both end users and stakeholders.
Overall, the proposed work seeks to deliver an accurate, robust, and real-
time pothole detection system that can effectively address the challenges associated
with road maintenance and infrastructure management. Through a systematic
approach to model development, evaluation, and optimization, the project aims to
achieve its objectives and contribute to improving road safety and infrastructure
maintenance efforts.

1.4 Organization of the Report

The report on the Realtime Pothole Detection System is systematically

organized to offer a comprehensive understanding of the project's development and
outcomes. It begins with an Introduction that outlines the project's significance,
followed by a Literature Survey that reviews existing work and relevant research in
the field of pothole detection systems. The Problem Statement section delves into
the specific challenges addressed by the project and the issues associated with
pothole detection.

The Methodology section provides a detailed account of the data collection

and preprocessing methods employed, the machine learning model chosen (SSD)
and its architecture, as well as the strategies used for model training,
hyperparameter optimization, and evaluation. Subsequently, the Results and
Analysis section presents and discusses the achieved results, comparing the model's
performance across different configurations and interpreting the key findings and
insights derived from the results.

3
 The report also includes a Future Scope section that explores potential
enhancements and extensions to the current system, discussing future research
directions and opportunities for improvement. The Conclusion summarizes the key
findings, contributions, and the project's significance, while the References section
lists the sources, references, and literature cited throughout the report. Lastly, the
Appendices contain supplementary material, additional data, code snippets, and
detailed documentation to support the main content of the report.

This organized structure ensures a logical flow of information, facilitating

readability, and easy navigation for readers to follow the project's development
process, understand the methodologies employed, and gain insights into the
project's outcomes and future prospects effectively.

4
 2. LITERATURE SURVEY

The literature survey conducted for the Realtime Pothole Detection System
project provided a comprehensive evaluation of the Single Shot Detector (SSD) in
the context of object detection, with a particular focus on its application in pothole
detection. Object detection plays a crucial role in various computer vision
applications, including road maintenance, safety, and infrastructure management.
SSD emerged as a leading object detection framework due to its efficiency,
accuracy, and suitability for real-time detection tasks, making it a prime candidate
for the project's objectives.

The survey commenced by emphasizing the significance of object detection

in road maintenance and safety applications, highlighting the challenges associated
with pothole detection and the potential impact on road users and infrastructure
[5][3][6].

Key performance indicators such as Mean Average Precision (mAP),

detection accuracy, processing speed, model complexity, and adaptability to
different environmental conditions were considered to evaluate the efficiency,
reliability, and robustness of SSD in real-world scenarios [11][10]. Comparative
analyses were conducted to assess SSD's performance against other object detection
frameworks, including YOLO (You Only Look Once) [10][11][15], to identify its
strengths, limitations, and trade-offs in pothole detection tasks [4][7].

The findings from the literature survey provided valuable insights into the
capabilities and potential of SSD in pothole detection applications. SSD
demonstrated superior performance in terms of detection accuracy, processing
speed, and adaptability to varying lighting and environmental conditions compared
to YOLO [2]. Its multi-scale feature extraction capabilities and efficient use of
computational resources enable SSD to detect potholes of various sizes, shapes, and
dimensions with higher precision and reduced false positives [13].

Research by [2] compares various object detection algorithms for pothole

detection. They found SSD to achieve a good balance between speed and accuracy,
making it suitable for real-time applications on resource-constrained edge devices.

5
Studies like [3] delve into optimizing pre-trained SSD models for pothole detection
tasks. This can involve techniques like transfer learning and fine-tuning the network
on pothole image datasets specific to your region or road conditions. This can lead
to a model that is both accurate in pothole detection and efficient in processing
power consumption.

While YOLO variants like YOLOv5 offer potentially higher accuracy, SSD
can be faster due to its single-stage processing pipeline [10][11][15]. This speed
advantage makes SSD more suitable for real-time applications where immediate
pothole detection is critical. Research on strategies for creating large and diverse
pothole image datasets specifically for your region or application can be valuable
[4]. This might involve collecting images under various lighting conditions, weather
scenarios, and different road surface textures to improve the model's
generalizability.
Firebase Realtime Database or Cloud Firestore can be leveraged for storing
pothole location data, including GPS coordinates, damage severity, and timestamps
[20]. These services allow for real-time updates whenever a new pothole is detected,
facilitating efficient data access for visualization and analysis.
Integrating pothole data with Google Maps allows users to visualize
detected potholes and plan their routes accordingly: Studies on integrating real-time
pothole detection data with Google Maps can provide insights on data visualization
and user interaction. Look for research on using Google Maps APIs to display
pothole information along user-defined routes or within user-specified areas [19].
This can empower users to make informed decisions about their journeys and
contribute to safer road conditions.
Limitations and challenges of real-time pothole detection systems. This can
include factors like varying lighting conditions, weather, shadows, and different
road surface textures that can impact detection accuracy [3][4][5][7]. Consider
strategies to mitigate these challenges, such as image pre-

By incorporating the insights from this literature survey, you can build a robust,
user-friendly, and scalable Real-Time Pothole Detection System with SSD and
Google Maps Integration. This system can contribute significantly to improving
road safety and facilitating efficient road maintenance efforts.

6
 3.PROPOSED WORK
3.1 Block Diagram

Figure (i) – Block diagram of proposed work

The block diagram in Figure – (i) delineates the sequential flow of

operations within the pothole detection system.
1. Dataset Collection: The system begins by collecting a video feed from an
ESP32 camera module.
2. Labelling the Images: Next, the images from the video feed are labelled
to identify damaged areas of the road.
3. Dataset Augmentation: In order to increase the size and variety of the
dataset, the labelled images are augmented. This can involve techniques like
flipping the images horizontally, cropping them, or adding noise.
4. Splitting the Dataset: The augmented dataset is then divided into three
parts: a training set, a test set, and a validation set. The training set is used to train
the machine learning model to detect damaged roads. The test set is used to evaluate
the performance of the trained model on unseen data. The validation set is used to
fine-tune the hyperparameters of the model during training.

7
 5. Object Detection and Location Tracking: Here, the machine learning
model is split into two parts:
Object detection: This part of the model is responsible for identifying and
locating damaged areas in the road images.
Location tracking: This part of the model uses a Neo6M GPS module to
track the location of the camera as it captures video.
6. Training the Dataset: The training set is fed into the object detection
model to train it to recognize damaged roads.
7. Obtain Results: Once trained, the model is used to process the test set and
identify damaged roads in the images.
8. Damaged Road Detected: If damaged road is detected, the Neo6M GPS
module logs the location of the damage and updates it in firebase.

Figure (ii) – Use Case Diagram

8
3.2 Algorithm

The Realtime Pothole Detection System project utilized the Single Shot
Detector (SSD) algorithm as the core machine learning model for object detection,
specifically targeting potholes on roads. SSD is a state-of-the-art object detection
algorithm known for its efficiency, accuracy, and suitability for real-time
applications.

Algorithm for preparing a custom dataset for object detection tasks using the
Single Shot Detector (SSD) architecture:

1. Import necessary libraries and modules

2. Define CustomDataset class:
2.1 Initialize with directory path, image dimensions, classes, and transforms
2.2 Read all image paths from directory
2.3 Implement __getitem__ method to:
2.3.1 Read image and preprocess (resize, normalize)
2.3.2 Read corresponding XML annotations
2.3.3 Adjust bounding box coordinates based on resized image dimensions
2.3.4 Return preprocessed image and dictionary with bounding box coordinates,
labels, area, etc.
2.4 Implement __len__ method to return the total number of images in the dataset
3. Define helper functions create_train_dataset, create_valid_dataset,
create_train_loader, create_valid_loader:
3.1 Instantiate CustomDataset with appropriate parameters
3.2 Create DataLoader with batch size, shuffle, collate function, etc.
4. If executed as main:
4.1 Instantiate CustomDataset for sanity check
4.2 Visualize a specified number of sample images with bounding boxes and labels

The SSD algorithm operates by generating a fixed set of default bounding

boxes across different feature maps and scales within the neural network. These
default boxes are then used to predict both the object's class and the offset to adjust
the default box to tightly fit the detected object. This approach enables SSD to

9
perform simultaneous object localization and classification in a single forward pass
through the network, hence the name "Single Shot."

Algorithm SSD Model Creation:

1. Import required libraries:

1.1 torchvision
1.2 torch.nn

2. Define create_model function:

2.1 Initialize backbone:
2.1.1 Load pretrained ResNet-34 from torchvision with ImageNet weights
2.1.2 Extract and organize ResNet-34 components into a sequential backbone

2.2 Initialize default box generator:

2.2.1 Define default box sizes and aspect ratios for each feature map layer
2.2.2 Configure different default box settings for various feature map layers

2.3 Initialize SSD head:

2.3.1 Define SSDHead class to predict class scores and bounding box offsets for each
default box across feature map layers
2.3.2 Set output channels from backbone layers and number of default boxes per
location

2.4 Initialize SSD model:

2.4.1 Combine backbone, default box generator, and SSD head to construct the SSD
model
2.4.2 Set model parameters including backbone, number of classes, anchor
generator, input size, head, and NMS threshold

2.5 Return SSD model

3. If __name__ == '__main__':
3.1 Call create_model function with parameters (num_classes=2, size=300) to create
SSD model instance

10
 3.2 Print SSD model architecture
3.3 Calculate and print total number of parameters and trainable parameters in the
model

The architecture of SSD comprises a base convolutional neural network

(CNN) followed by several auxiliary convolutional layers responsible for detecting
objects at multiple scales. These auxiliary layers are designed to capture features at
different resolutions, allowing SSD to detect objects of various sizes and aspect
ratios effectively. The SSD algorithm utilizes a combination of classification and
regression losses during training to optimize the model's parameters and improve
its detection performance.

Algorithm for training an object detection model using the Single Shot
Detector (SSD) architecture:

1. Import necessary modules and functions from:

1.1 config
1.2 model
1.3 custom_utils
1.4 datasets
1.5 torch
1.6 matplotlib.pyplot
1.7 time
1.8 os

2. Set seed for reproducibility

3. Define create_model function:

3.1 Load pretrained ResNet34 backbone
3.2 Create SSD model with backbone, anchor generator, and SSDHead
3.3 Return SSD model

4. Define train function taking train_data_loader and model as arguments:

4.1 Set model to training mode

11
 4.2 Initialize tqdm progress bar
4.3 For each batch in train_data_loader:
4.3.1 Forward pass through model
4.3.2 Compute losses
4.3.3 Accumulate losses
4.3.4 Backward pass and update model parameters
4.3.5 Update tqdm progress bar description with current loss
4.4 Return final loss value

5. Define validate function taking valid_data_loader, model, and device as arguments:

5.1 Set model to evaluation mode
5.2 Initialize MeanAveragePrecision metric
5.3 Initialize lists for storing true and predicted bounding boxes and labels
5.4 For each batch in valid_data_loader:
5.4.1 Forward pass through model to get predictions
5.4.2 Extract true and predicted bounding box and label information
5.4.3 Append information to respective lists
5.5 Compute mAP using MeanAveragePrecision metric
5.6 Return computed mAP value

6. If __name__ == '__main__':
6.1 Create directories for outputs if not exist
6.2 Load training and validation datasets and create data loaders
6.3 Initialize and move model to device (CPU or GPU)
6.4 Print total and trainable parameters of the model
6.5 Initialize optimizer and learning rate scheduler
6.6 Initialize loss and mAP history trackers

6.7 If visualize transformed images flag is True:

6.7.1 Visualize transformed images from train_loader

6.8 For each epoch in range (NUM_EPOCHS):

6.8.1 Reset training loss history
6.8.2 Start timer

12
 6.8.3 Run training and validation functions
6.8.4 Print epoch, training loss, and mAP values
6.8.5 Save best model based on mAP
6.8.6 Save current epoch model
6.8.7 Save loss and mAP plots
6.8.8 Step learning rate scheduler

In the context of the Realtime Pothole Detection System project, the SSD
algorithm was trained using a custom dataset of road images annotated with
bounding boxes around potholes. The training process involved optimizing key
hyperparameters, such as the learning rate, batch size, number of epochs, and loss
function, to ensure efficient model convergence and performance. Various data
augmentation techniques, including random cropping, grayscale conversion,
brightness and contrast adjustments, were also employed to enhance the model's
generalization and robustness.

Once trained, the SSD model was deployed to perform real-time pothole
detection using live camera feeds integrated with the system. The detected potholes
were then visualized and analyzed on a comprehensive website integrated with
Google Maps API, providing users with a detailed overview of pothole locations,
sizes, and other relevant information for road maintenance and safety purposes.

Overall, the SSD algorithm's architecture, efficiency, and real-time

processing capabilities made it an ideal choice for the Realtime Pothole Detection
System project, enabling accurate and efficient detection of potholes under various
lighting and environmental conditions, thereby contributing to improved road safety
and infrastructure maintenance.

13
 4. EXPERIMENTAL STUDY
4.1 Data Sets
In the Realtime Pothole Detection System project, the dataset plays a crucial
role in training and evaluating the performance of the machine learning models.
Here's an explanation of the datasets used in the project, based on the provided
information:

The dataset for the project was sourced from Kaggle [15][16][17][18], a
platform known for hosting machine learning datasets and competitions. The
dataset comprises of 768 unique images containing various road conditions and
pothole instances.

To facilitate the training of the computer vision model, the images in the
dataset were annotated using LabelMe, a popular tool for annotating images with
bounding boxes. This annotation process involved marking the location of potholes
and other relevant objects or features in the images. The annotated data, which
contains information about the coordinates and dimensions of the bounding boxes,
was then stored in XML format. This structured data is essential for training the
machine learning models to recognize and detect potholes accurately.

To enhance the diversity and robustness of the dataset, image augmentation

techniques were applied using the Albumentations library. These augmentation
techniques included: Random Cropping, Grayscale Conversion, Saturation
Adjustment, Brightness Adjustment, Hue Adjustment, Shadow Adjustment,
Nighttime Simulation

By applying these augmentation techniques, the dataset was diversified,

exposing the model to a wider range of scenarios and enhancing its ability to
generalize and perform well in real world conditions.

In summary, the dataset used in the project consists of over 700 annotated
images sourced from Kaggle, which were further enhanced and diversified through
image augmentation techniques using the Albumentations library. This curated and

14
augmented dataset serves as the foundation for training and evaluating the machine
learning models developed for the Realtime Pothole Detection System.

Figure (iii) – Dataset after applying Augmentation Techniques

4.2 Software Requirements

The software requirements for the Realtime Pothole Detection System
project are diverse, spanning from core programming languages and libraries to
hardware interfacing and web development tools.
Python 3.10+ is the foundational programming language, facilitating
machine learning and image processing tasks.
OpenCV is essential for computer vision and image analysis, supporting
pothole detection and feature extraction.
LabelMe is used for image annotation, creating bounding box annotations
for training the machine learning models.
Arduino IDE is employed for hardware interfacing, enabling real-time data
collection with sensors or GPS modules.
Visual Studio Code serves as the primary development environment, aiding
in coding, debugging, and managing the project's source code.
The PyTorch, Torch, TorchVision, and Albumentation packages form
the machine learning pipeline, with PyTorch providing the framework for model
development and training, and Albumentation supporting dataset augmentation.
A PC with Windows/Linux operating system is required to run the
software components.

15
 Google Colab Account offers a cloud-based platform for GPU accelerated
model training.
ReactJS and NodeJS are used for web development, creating the project's
frontend and backend components.
Firebase provides real-time database and authentication services.
Google Maps API enables geolocation functionalities.
Lastly, a browser with JavaScript and CORS enabled is essential for web
interface interaction and compatibility.

4.3 Hardware Requirements

Figure (iv): Neo6M GPS Module interfacing with ESP32

PC with Nvidia Graphics card (if Colab is not used)

A PC equipped with an Nvidia Graphics card is recommended to handle the
computational demands of training complex machine learning models locally. The
GPU accelerates the training process by performing parallel processing,
significantly reducing the time required for model training compared to CPUonly
computations. This hardware component is essential for running intensive tasks
such as model training and optimization, ensuring efficient and timely development
of the Realtime Pothole Detection System.
5GB+ Storage

16
 A minimum of 5GB of storage is required to accommodate the project's
datasets, software tools, libraries, and other resources. Adequate storage space
ensures smooth data handling, model training, and software development processes
without the risk of running out of space, facilitating seamless progress and efficient
project management.

ESP32 Camera Module

Figure (v): ESP32 Camera Module

The ESP32 Camera Module serves as the hardware component responsible

for capturing high-quality images or video footage of road surfaces. Integrated with
an ESP32 microcontroller, this camera module enables real-time data collection of
road conditions, capturing images or video feeds that are subsequently processed
by the machine learning models for pothole detection. The ESP32 Camera Module's
compact design, high-resolution capabilities, and compatibility with various
interfaces make it an ideal choice for capturing detailed and reliable data for the
Realtime Pothole Detection System.
Neo6M GPS Module

Figure (vi): Neo6M GPS Module

17
 The Neo6M GPS Module is an essential hardware component that facilitates
geolocation data collection by capturing geographical coordinates and locations of
detected potholes. Integrated with the system, this GPS module provides accurate
positioning information, enabling the system to identify and map the exact locations
of potholes on a geographical platform. The Neo6M GPS Module's high accuracy,
reliability, and compatibility with the project's hardware and software components
make it a crucial tool for enhancing the system's precision and effectiveness in real-
time pothole detection and mapping.
In summary, the hardware requirements for the Realtime Pothole Detection
System project encompass a range of essential components, including a powerful
PC with a Nvidia Graphics card for computational tasks, sufficient storage for data
management, ESP32 Camera Module for real-time image capture, and Neo6M GPS
Module for accurate geolocation data collection. These hardware components play
a pivotal role in enabling data collection, system integration, and real-time
processing, facilitating the development and deployment of an efficient and scalable
Realtime Pothole Detection System.

4.4 Preprocessing

1. Data Collection:

Purpose: The primary objective of data collection is to gather high quality

images or video footage of road surfaces where potholes are likely to occur.

Process: The ESP32 Camera Module is utilized to capture images or video feeds
of road surfaces in different environmental conditions, such as varying lighting,
weather, and road types.

Importance: High resolution and diverse data collection ensures that the
machine learning models are exposed to various scenarios and conditions,
enhancing their ability to generalize and accurately detect potholes in real-world
settings.

2. Image Annotation:

18
 Purpose: Image annotation involves labelling the collected images by marking
the locations of potholes and other relevant objects using bounding boxes.

Process: The LabelMe tool is employed to manually annotate the images,

specifying the coordinates and dimensions of potholes and other objects.

Importance: The annotated images provide ground truth data required for
supervised learning, enabling the machine learning models to learn and recognize
potholes based on the labelled examples, thereby improving detection accuracy and
reducing false positives.

3. Data Augmentation:

Purpose: Data augmentation techniques are applied to increase the diversity and
robustness of the dataset, enhancing the model's generalization capabilities.

Process: The Albumentation library is used to apply various augmentation

techniques, such as random cropping, grayscale conversion, saturation adjustment,
brightness adjustment, hue adjustment, shadow adjustment, and nighttime
simulation.

Importance: Augmentation introduces variability, handles colour variations,

adjusts brightness, simulates different lighting conditions, accounts for diverse
scenarios, and ensures model adaptability, making the machine learning models
more resilient to variations and anomalies encountered in real-world data.

4. Data Splitting:

Purpose: Data splitting involves partitioning the annotated and augmented

dataset into training, validation, and testing sets to facilitate model training,
hyperparameter optimization, and performance evaluation.

Process: The dataset is divided into mutually exclusive subsets, typically in a

ratio such as 70% for training, 15% for validation, and 15% for testing.

Importance: Data splitting ensures unbiased evaluation and validation of the

machine learning models, preventing overfitting, and providing insights into the

19
model's performance on unseen data, enabling iterative refinement and
improvement.

5. Normalization and Standardization:

Purpose: Normalization and standardization are applied to ensure consistent and

uniform data representation across the dataset, facilitating optimal model training
and convergence.

Process: The pixel values of the pre-processed images are normalized and
standardized to a common scale or range, such as [0, 1] or [1, 1].

Importance: Consistent data representation reduces the variance in feature

scales, accelerates model convergence, improves model stability, and enhances the
model's ability to learn meaningful patterns and relationships, ultimately leading to
improved detection accuracy and performance.

6. Data Integration:

Purpose: Data integration involves combining the annotated images with

geolocation data collected from the Neo6M GPS Module to associate geographical
coordinates and locations with detected potholes.

Process: The geographical coordinates obtained from the GPS module are
mapped to the corresponding annotated images, creating a spatial reference for each
detected pothole.

Importance: Data integration enables the visualization, mapping, and analysis

of pothole locations on a geographical platform, enhancing the system's utility,
effectiveness, and user experience by providing actionable insights, facilitating
targeted maintenance, and supporting informed decision making.

In summary, the preprocessing steps in the Realtime Pothole Detection System

project encompass meticulous data collection, image annotation, data
augmentation, data splitting, normalization and standardization, and data
integration processes. These preprocessing steps are essential for preparing the raw

20
data for machine learning model training, ensuring data quality, diversity,
consistency, relevance, and spatial context, ultimately facilitating the development
and deployment of an accurate, efficient, and scalable Realtime Pothole Detection
System capable of identifying, mapping, and addressing potholes in real time.

4.5 Parameter Setting

1. Hyperparameters:

Learning Rate: After extensive experimentation, a learning rate of 0.0005 was

determined to be optimal for efficient model convergence and performance.

Batch Size: A batch size of 16 was selected based on available memory and
computational resources, balancing training speed and memory usage.

Number of Epochs: The model was trained for 75 epochs to ensure adequate
learning without overfitting to the training data.

Optimizer: SGD with learning rate 0.0005, momentum 0.9, and Nesterov
momentum.

Loss Function: SSD internally uses a combined loss function addressing both
classification and localization tasks. Classification loss: Measures how well the
model classifies objects within bounding boxes. Localization loss: Measures the
difference between predicted and ground truth bounding boxes.

Activation Function: ReLu is a non-linear activation function commonly used in

neural networks. It introduces non-linearity, allowing the network to learn more
complex relationships between features in the data.

2. Model Architecture:

Backbone Architecture: ResNet variants, specifically ResNet32, were chosen as

the backbone architecture for feature extraction, leveraging their proven
effectiveness in object detection tasks with large feature sets.

21
 Object Detection Framework: SSD (Single Shot Detector) was employed for
predicting bounding boxes at different scales in a single pass through the network,
facilitating efficient and accurate object detection.

Figure (vii): SSD Model Architecture

SSD, or Single Shot Detector, is a popular object detection algorithm that

has gained significant attention for its efficiency and effectiveness in various
computer vision tasks, including real time object detection. Developed by Wei Liu
et al., SSD revolutionized object detection by addressing some of the limitations of
earlier models like Faster RCNN in terms of speed and accuracy.

The key innovation of SSD lies in its ability to perform object detection in
a single forward pass through a neural network, eliminating the need for a two-stage
process as seen in previous models. This approach significantly speeds up the
detection process, making SSD highly suitable for real time applications where
speed is crucial.

Architecture:

SSD's architecture consists of a base convolutional neural network (CNN),

typically based on VGG16 or ResNet, followed by a series of additional
convolutional layers. These layers are responsible for predicting bounding boxes
and class scores at multiple scales within the image. By using multiple feature maps
at different scales, SSD is capable of detecting objects of various sizes and aspect
ratios, ensuring comprehensive coverage of the image.

Multiscale Feature Maps:

22
 One of the distinguishing features of SSD is its use of multiscale feature
maps to predict bounding boxes at different resolutions. This enables SSD to detect
objects of various sizes effectively, from small to large, within a single network
architecture. Each feature map is associated with default bounding boxes of
different sizes and aspect ratios, allowing SSD to detect objects with high precision
across the entire image.

Default Boxes:

SSD uses a set of default boxes or anchor boxes at different aspect ratios
and scales to predict the location and size of objects within the image. These default
boxes are predefined and serve as starting points for the detection process, which is
refined based on the predictions made by the network. By utilizing default boxes at
multiple scales and aspect ratios, SSD achieves high detection accuracy and
robustness against objects of different sizes and orientations.

Prediction Layers:

SSD incorporates prediction layers at various scales in the network

architecture to generate class scores and adjust the positions of default boxes. These
prediction layers are responsible for identifying the presence of objects within the
image and refining the bounding box coordinates to accurately enclose the detected
objects.

Loss Function:

SSD employs a combination of localization loss and confidence loss to train

the network effectively. The localization loss measures the discrepancy between the
predicted bounding box coordinates and the ground truth coordinates, while the
confidence loss evaluates the classification accuracy of the predicted class scores.
By optimizing both losses simultaneously, SSD learns to accurately localize and
classify objects within the image, achieving high detection performance.

3. Data Augmentation Parameters:

23
 Augmentation Techniques: A comprehensive set of augmentation techniques,
including random cropping, grayscale conversion, saturation adjustment, brightness
adjustment, hue adjustment, shadow adjustment, and nighttime simulation, were
applied to enhance model robustness and generalization capabilities.

4. Regularization Parameters:

Dropout Rate: A dropout rate of 0.5 was applied during training to prevent
overfitting by randomly deactivating neurons and promoting model generalization.

Weight Decay: A weight decay term with a value of 0.0001 was added to the loss
function to penalize large weights and prevent overfitting.

5. Evaluation Metrics:

[email protected]: The mean Average Precision calculated at an IoU (Intersection over

Union) threshold of 0.5 was used as the primary evaluation metric, achieving an
[email protected] value of 88.676%, indicating high detection performance.

Precision, Recall, and F1 Score: Precision, Recall, and F1 Score metrics were
computed, demonstrating the model's ability to correctly detect potholes, identify
all potholes, and balance Precision and Recall effectively.

4.6 Results

Figure(viii): Detection of Potholes in Different Lighting Conditions by our SSD

model

The developed model demonstrated robust performance in detecting potholes

under various lighting conditions, including nighttime. This indicates the model's

24
adaptability and reliability in real-world scenarios where lighting can vary
significantly.

Figure (ix): Detection of Potholes of Different Sizes, Shapes, and Dimensions

The model showcased the ability to detect potholes of diverse sizes, shapes, and
dimensions with high accuracy. This versatility ensures comprehensive coverage
and effective detection of potholes across different road surfaces and environments.

Figure(x): Detection of False Objects like Trees, Shadows, Vehicles, Healthy

Roads

The model successfully addressed the challenge of false object detections by

accurately distinguishing between potholes and other objects such as trees,
shadows, vehicles, etc. This enhances the model's precision and minimizes false
alarms, ensuring reliable detection results.

25
 Figure(xi): Comprehensive website integrated with Google Maps API for
analyzing results and viewing potholes in a route

The comprehensive website serves as the central interface where users can
access and analyze the pothole detection results. It offers a user-friendly
environment with intuitive navigation and interactive features, allowing users to
easily explore and interpret the collected data.

The integration with Google Maps API enables the visualization of detected
potholes directly on the map interface, providing spatial context and geographical
distribution of the identified potholes. Users can view the locations of detected
potholes, zoom in/out for detailed analysis, and click on individual markers to
access additional information about each pothole, such as size, shape, and severity.

The website facilitates the analysis of pothole detection results by displaying

them in a structured and organized manner. Users can filter and sort the detected

26
potholes based on various parameters, such as location, size, and detection
confidence, to identify trends, patterns, and areas requiring attention.

The real-time integration with Google Maps API ensures that the website's
map interface reflects the latest pothole detection results, providing users with up-
to-date information and enabling prompt action and response to identified issues.

4.7 Analysis:

Table (I): Results obtained while training SSD model under various parameters

1. Momentum Variation:

Momentum values of 0.5 and 0.9 were tested. A momentum value of 0.9 yielded
the highest mAP@50 of 85.85, indicating that higher momentum contributes to
better detection performance. However, a momentum of 0.5 resulted in a
significantly lower mAP@50 of 13.15, suggesting that too low a momentum value
may adversely affect detection accuracy.

2. Learning Rate Variation:

Learning rates ranged from 0.0000002 to 0.01. The highest mAP@50 of

85.85 was achieved with a learning rate of 0.01. A learning rate of 0.002 with
ADAM resulted in the lowest mAP@50 of 13.15, indicating that an extremely low
learning rate is detrimental to model performance. Optimal performance seems to
be achieved with a moderate learning rate around 0.01.

3. Optimizer Variation:

27
 Two types of optimizers, SGD (Stochastic Gradient Descent) and ADAM
(Accelerated Gradient Descent), were tested. SGD yielded the highest mAP@50 of
88.676 and 85.85, with momentum values of 0.1 and 0.9, respectively. AGD
resulted in lower mAP@50 values ranging from 3.5 to 49.9 across different
configurations. SGD appears to be more effective in optimizing the model for
higher detection accuracy compared to AGD in this context.

4. Train Loss Analysis:

The achieved train loss values varied across different configurations. Lower train
loss values generally corresponded to higher mAP@50 values, indicating a
correlation between lower train loss and better model performance. For instance, a
train loss of 0.545 resulted in a mAP@50 of 25.769, while a train loss of 0.6764
yielded a mAP@50 of 88.676, reinforcing the importance of minimizing training
loss for optimal performance.

Higher momentum values and moderate learning rates around 0.01 appear
to contribute to better detection performance. The SGD Optimizer outperformed the
AGD Optimizer in optimizing the model for higher detection accuracy. Lower train
loss values are indicative of better model performance, highlighting the importance
of effective model training and optimization strategies.

Figure(xii): Iterations vs Train Loss and Epoch vs mAP@50

The above analysis is done for momentum 0.1 with learning rate of 0.0005 having
SGD (Stochastic Gradient Descent) optimizer achieving 88.676 mAP@50 and train
loss of 0.6764.

28
 Figure(xiii): Iterations vs Train Loss and Epoch vs mAP@50

The above analysis is done for momentum 0.5 with learning rate of 0.0000002
having SGD (Stochastic Gradient Descent) optimizer achieving 13.15 mAP@50
and train loss of 39.944

Figure(xiv): Iterations vs Train Loss and Epoch vs mAP@50

The above analysis is done for momentum 0.9 with learning rate of 0.01 having
SGD (Stochastic Gradient Descent) optimizer achieving 85.85 mAP@50 and train
loss of 0.9760

29
 Figure(xiv): Iterations vs Train Loss and Epoch vs mAP@50

The above analysis is done for momentum 0.9 with learning rate of 0.001 having
AGD (Accelerated Gradient Descent) optimizer achieving 49.90 mAP@50 and
train loss of 1.892

Figure(xvi): Iterations vs Train Loss and Epoch vs mAP@50

The above analysis is done for momentum 0.1 with learning rate of 0.001 having
ADAM optimizer achieving 25.769 mAP@50 and train loss of 0.5450

Figure(xvii): Iterations vs Train Loss and Epoch vs mAP@50

30
The above analysis is done for momentum 0.1 with learning rate of 0.01 having
ADAM optimizer achieving 14.2 mAP@50 and train loss of 5.026

Figure(xviii): Iterations vs Train Loss and Epoch vs mAP@50

The above analysis is done for momentum 0.1 with learning rate of 0.02 having
ADAM optimizer achieving 3.5 mAP@50 and train loss of 7.5

The data with a momentum of 0.5, a learning rate of 0.0000002, and an SGD
optimizer produced a notably lower mAP@50 of 13.15 with a train loss of 39.944.
This configuration may be less favourable due to the extremely low learning rate.
With a momentum of 0.1, a learning rate of 0.01, and an SGD Optimizer, the results
showed a mAP@50 of 88.676 and a train loss of 0.6764. This suggests that higher
momentum and a moderate learning rate contribute to better detection
performance.

Using a momentum of 0.9, a learning rate of 0.001, and an ADAM optimizer

resulted in a mAP@50 of 49.9 with a train loss of 1.892. This indicates that the
ADAM optimizer might be less effective compared to the SGD Optimizer in this
scenario. The configuration with a momentum of 0.1, a learning rate of 0.001, and
an ADAM optimizer yielded a mAP@50 of 25.769 and a train loss of 0.545. This
suggests that a lower momentum combined with a moderate learning rate and
ADAM optimizer may not be as effective for achieving high mAP@50 values.

31
 5. CONCLUSION AND FUTURE SCOPE

The Realtime Pothole Detection System project was undertaken with the
primary objective of developing an efficient machine learning model capable of
real-time detection of potholes. To achieve this, the project focused on leveraging
the capabilities of the SSD (Single Shot Detector) model. A series of optimization
processes were conducted to finetune key hyperparameters, including the learning
rate, batch size, number of epochs, optimizer, and loss function. As a result of these
optimizations, specific values were determined for the hyperparameters, namely a
learning rate of 0.0005, a batch size of 16, and training for 75 epochs, SGD
Optimizer and with default Loss function.

In terms of data preparation and model training, a custom dataset sourced

from Kaggle was utilized as the foundation for training the model. To enhance the
model's generalization capabilities and robustness against various real-world
scenarios, several image augmentation techniques were implemented during the
preprocessing stage. These techniques played a crucial role in diversifying the
dataset and exposing the model to a wider range of potential scenarios, ultimately
contributing to its improved performance and adaptability.

The implementation of the developed Realtime Pothole Detection System

yielded notable improvements in pothole detection across different environmental
conditions and effectively addressed issues related to false object detections,
enhancing the overall reliability and accuracy of the system.

Looking ahead, the project has several avenues for future development and
enhancement. Firstly, there is a need for further optimization of the model to
enhance its accuracy, efficiency, and real-time performance, potentially through
additional finetuning of hyperparameters or exploring advanced model
architectures. Secondly, improvements in real-time camera feed handling and
integration with IP streaming solutions could be pursued to bolster the system's
detection capabilities and responsiveness. Additionally, the integration of the
system with the Google Maps API could facilitate location-based analysis and
visualization of detected potholes, offering valuable insights for infrastructure
maintenance and planning.
32
 Continued refinement of performance metrics, such as mAP@50, precision,
recall, and overall accuracy, will be essential to achieving higher standards and
reliability in pothole detection. Furthermore, retraining the model with an expanded
and diversified dataset comprising 5000+ augmented images could be undertaken
to further enhance its detection capabilities. Lastly, the development of a
comprehensive user interface integrated with the Google Maps API would enable
users to analyse, visualize, and report detected potholes and damages in real-time,
enhancing user engagement and system usability.

33
 REFERENCES
[1]. Optimal Fuzzy Wavelet Neural Network Based Road Damage Detection (June
2023). Mohammed Alamgeer, Hend Khalid Alkahtani, Mashael Maashi.
[2]. Au Yang Her, Weng Kean Yew, Pang Jia Yew, Melissa Chong Jia Ying.
Realtime pothole detection system on vehicle using improved YOLOv5 in Malaysia
((2022).
[3] Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE. A survey of deep neural
network architectures and their applications. Neurocomputing. (2020)
[4] Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS, Hasan M,
Van Essen BC, Awwal AA, Asari VK. A stateoftheart survey on deep learning
theory and architectures. Electronics. (2021)
[5] Y. Darma, M. R. Karim, and S. Abdullah, “An analysis of Malaysia road traffic
death distribution by road environment,” Sadhana Academy Proceedings in
Engineering Sciences, vol. 42, no. 9, pp. 1605–1615, Sep. 2017, doi:
10.1007/s120460170694 9.
[6] “Malaysia’s Minister Khairy Jamaluddin injures from fall after bicycle hits
pothole,” The Straits Times, Dec. 28, 2020. Accessed: May 31, 2022. [Online].
Available:
https://www.straitstimes.com/asia/seasia/malaysiasministerkhairyjamaluddininjure
dfromfallafterbicyclehitspothole
[7] K. Perimbanayagam, “75yearold man killed after crashing into pothole,” New
Straits Times, Jan. 03, 2021. Accessed: May 31, 2022. [Online]. Available:
https://www.nst.com.my/news/nation/2021/01/654169/75yearoldmankilledaftercr
ashingpothole
[8] S. Thiruppathiraj, U. Kumar, and S. Buchke, “Automatic pothole classification
and segmentation using android smartphone sensors and camera images with
machine learning techniques,” in 2020 IEEE REGION 10 CONFERENCE
(TENCON), Nov. 2020, pp. 1386–1391. doi:
10.1109/TENCON50793.2020.9293883.
[9] R. Fan, U. Ozgunalp, B. Hosking, M. Liu, and I. Pitas, “Pothole Detection
Based on Disparity Transformation and Road Surface Modeling,” IEEE
Transactions on Image Processing, vol. 29, pp. 897–908, 2020, doi:
10.1109/TIP.2019.2933750.

34
[10] J. W. Baek and K. Chung, “Pothole classification model using edge detection
in road image,” Applied Sciences (Switzerland), vol. 10, no. 19. MDPI AG, Oct.
01, 2020. doi: 10.3390/APP10196662.
[11] Y. Jo and S. Ryu, “Pothole detection system using a blackbox camera,” Sensors
(Switzerland), vol. 15, no. 11, pp. 29316–29331, Nov. 2015, doi:
10.3390/s151129316.
[12] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path Aggregation Network for Instance
Segmentation,” Mar. 2018.
[13] GitHub, “GitHub ultralytics/yolov5.” https://github.com/ultralytics/yolov5
(accessed May 31, 2022).
[14] https://docs.ultralytics.com/models/#featuredmodels
[15] https://www.kaggle.com/datasets/atulyakumar98/potholedetectiondataset
[16] https://www.kaggle.com/datasets/andrewmvd/potholedetection
[17] https://www.kaggle.com/datasets/rajdalsaniya/potholedetectiondataset
[18]https://www.kaggle.com/datasets/prudhvignv/roaddamageclassificationandass
essment
[19] https://developers.google.com/maps/documentation/
[20] Firebase Documentation (google.com)

35