Explainable and Trustworthy Traffic Sign Detection for

Safe Autonomous Driving: An Inductive Logic

Programming Approach
CAQ 413 Seminar

Seminar Report submitted in partial fulfillment for the degree

B.Tech in Computer Science and Engineering (Artificial Intelligence)

Submitted by

Anandhu E S(Reg No: MUT20CA012)

DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND DATA SCIENCE

(Affliated to APJ Abdul Kalam Technological University)

November 2023
 Varikoli P.O., Puthencruz-682308

DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND DATA SCIENCE

CERTIFICATE

This is to certify that the seminar report entitled ”Explainable and Trustworthy Traffic Sign De-
tection for Safe Autonomous Driving: An Inductive Logic Programming Approach” submitted by
Anandhu E S (Reg no: MUT20CA012) of Semester VII is a bonafide account of the work
done by her under our supervision during the academic year 2023-24

Guide Coordinator Head of the Department

Dr. Priya C V Ms. Jayalakshmi J Dr. Sumam Mary Idicula
Asst. Professor Asst. Professor Professor
Dept. of AI & DS Dept. of AI & DS Dept. of AI & DS
 Acknowledgements

This seminar is the result of my hard work wherein I have been helped and supported by several
persons and institutions directly and indirectly. Now it is the time to acknowledge their contribu-
tions.
Foremost, I would like to express my sincere gratitude to my project guide, Asst. Prof.
Dr.Priya C V for her continuous encouragement, invaluable guidance, motivation and enthusiasm
throughout my seminar. I appreciate her sincere help in terms of patience, time and ideas so as to
make my seminar experience stimulating and productive.
My sincere gratitude to Dr. Sumam Mary Idicula, Head of Department during my course
period 2020-2024. I would also like to thank the project coordinator Asst. Prof. Jayalekshmi
J for critically assessing my work and giving valuable suggestions. I would like to acknowledge
management of Muthoot Institute of Technology & Science for providing academic support to
complete my seminar.
In my daily work I have been blessed with a friendly and cheerful group of fellow students. I
am very much thankful to all the members of S7 CS(AI).
I would like to express my sincere gratitude to the technical staff of Deep Learning labora-
tory and all the staff of Artificial Intelligence and Data Science Department for providing good
environment.
Besides this, several people have knowingly and unknowingly helped me in the successful com-
pletion of this seminar. I express my sincere gratitude to all of them.

Anandhu E S
 Abstract

Traffic sign detection is a critical task in the operation of Autonomous Vehicles (AV), as it ensures
the safety of all road users. Current DNN-based sign classification systems rely on pixel-level
features to detect traffic signs and can be susceptible to adversarial attacks. These attacks involve
small, imperceptible changes to a sign that can cause traditional classifiers to misidentify the sign.
We propose an Inductive Logic Programming (ILP) based approach for stop sign detection in AVs
to address this issue. This method utilises high-level features of a sign, such as its shape, colour,
and text, to detect categories of traffic signs. This approach is more robust against adversarial
attacks, as it mimics human-like perception and is less susceptible to the limitations of current
DNN classifiers. We consider two adversarial attacking methods to evaluate our approach: Robust
Physical Perturbation (PR2) and Adversarial Camouflage (AdvCam). These attacks are able to
deceive DNN classifiers, causing them to misidentify stop signs as other signs with high confidence.
The results show that the proposed ILP-based technique is able to correctly identify all targeted
stop signs, even in the presence of PR2 and ADvCam attacks. The proposed learning method
is also efficient as it requires minimal training data. Moreover, it is fully explainable, making it
possible to debug AVs.
Contents

List of Figures iii

List of Abbreviations iv

1 Introduction 1

2 Literature Review 3
2.1 Abhishek Balasubramaniam and Sudeep Pasricha, Object Detection in Autonomous
Vehicles: Status and Open Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Gene Lewis, Object Detection for Autonomous Vehicles . . . . . . . . . . . . . . . . 3
2.3 Ross Greer, Akshay Gopalkrishnan, Nachiket Deo, Akshay Rangesh, Mohan Trivedi,
SALIENT SIGN DETECTION IN SAFE AUTONOMOUS DRIVING: AI WHICH
REASONS OVER FULL VISUAL CONTEXT . . . . . . . . . . . . . . . . . . . . . 4
2.4 Yizhe Wang , Xiaoguang Yang , Hailun Liang , and Yangdong Liu,A Review of the
Self-Adaptive Traffic Signal Control System Based on Future Traffic Environment . . 5
2.5 Luca Mora , Xinyi Wu, Anastasia Panori,Mind the gap: Developments in autonomous
driving research and the sustainability challenge . . . . . . . . . . . . . . . . . . . . 6

3 Methodology 7
3.1 Inductive Logical Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.2 General Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.3 Logical Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.4 Inductive Logic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.5 Hypothesis Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.6 Adversarial Attack Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4 Result and Discussions 12

i
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach Contents
5 Applications 14

6 Conclusion 16

7 Bibliography 18

A Presentation Slides 19

B Sample Reference Papers 23

C Viva questions and answers 28

B.Tech CS(AI) 2020-24 ii

List of Figures

Figure 3.1: ILP- based traffic sign classifier . . . . . . . . . . . . . . . . . . . . . . . . . 8

Figure 3.2: The general architecture flow . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Figure 3.3: Base data set for training and testing . . . . . . . . . . . . . . . . . . . . . 10
Figure 3.4: Base data set for training and testing . . . . . . . . . . . . . . . . . . . . . 11

Figure 4.1: Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

iii
List of Abbrevations

AI Artificial Intelligence
CNN Convolutional Neural Network
DNN Deep Neural Network
ILP Inductive Logical Programming
AV Autonomous vechicles
AdvCam Adversarial Camera
RP 2 Robust Physical Perturbation 2
LBP Local Binary Pattern
SVM Support Vector Machine
HSV Hue Saturation Value
HSL Hue Saturation Lightness
ROI Region of Interest
MSE Mean Squared Error

iv
Chapter 1

Introduction

”Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving” is a research
paper that proposes a new approach for traffic sign detection in Autonomous Vehicles (AVs) using
Inductive Logic Programming (ILP). The paper addresses the limitations of current Deep Neural
Network (DNN)-based traffic sign classifiers, which require a large amount of data for training,
are vulnerable to adversarial attacks, and are not explainable. The proposed ILP-based approach
mimics human traffic sign detection and uses high-level features of a sign, such as its shape, colour,
and text, for better interpretability and explainability.
The paper begins by discussing the importance of traffic sign detection in AVs and the challenges
associated with it. The authors highlight the fact that traffic sign detection is a critical task in
the operation of AVs, as it ensures the safety of all road users. They also point out that current
DNN-based approaches have several limitations, including the need for a large amount of data for
training, vulnerability to adversarial attacks, and lack of interpretability and explainability. The
authors argue that a more robust and explainable approach is needed to address these limitations.
The paper then introduces the concept of ILP and explains how it can be used for traffic sign
detection. The proposed approach uses a set of logical rules to detect traffic signs based on their
high-level features. The authors argue that this approach is more robust against adversarial attacks
and provides better interpretability and explainability than DNN-based approaches.
They also provide a detailed description of the ILP-based approach and explain how it works.
The paper presents experimental results that demonstrate the effectiveness of the proposed ap-
proach in detecting stop signs in various scenarios. The authors compare the proposed approach
with the traditional DNN-based approach and show that the ILP-based approach outperforms the
DNN-based approach in terms of robustness and interpretability. The proposed approach is data-
efficient, explainable, and able to withstand adversarial attacks that cannot easily deceive humans.
The authors conclude that the proposed approach can provide a more trustworthy and safe solution
for traffic sign detection in AVs.
The paper is well-written, and the arguments are well-supported by evidence. The authors

1
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 1. Introduction
have made a valuable contribution to the field of AVs by proposing a new approach for traffic sign
detection that addresses the limitations of current DNN-based approaches. The proposed approach
is more robust, explainable, and trustworthy, which is essential for the safe operation of AVs.
The paper provides a detailed description of the proposed approach, experimental results, and a
discussion of the implications of the findings. The authors also highlight the potential applications
of the proposed approach in other areas, such as medical diagnosis and fraud detection. Overall,
the paper is an important contribution to the field of AVs, and it is likely to inspire further research
in this area.

B.Tech CS(AI) 2020-24 2

Chapter 2

Literature Review

2.1 Abhishek Balasubramaniam and Sudeep Pasricha, Object De-

tection in Autonomous Vehicles: Status and Open Challenges
This paper discusses the landscape of various object detectors being considered and deployed in
emerging autonomous vehicles (AVs), the challenges involved in using these object detectors in
AVs, and how the object detectors can be optimized for lower computational complexity and faster
inference during real-time perception. The paper also presents a multitude of open challenges and
opportunities to advance the state-of-the-art with object detection for AVs. The paper highlights
the importance of object detection in autonomous driving and the challenges that come with it, such
as the resource-constrained nature of the onboard embedded computers used in vehicles, which have
limited memory availability and reduced processing capabilities due to stringent power caps, and
high susceptibility to faults due to thermal hotspots and gradients, especially during operation in
the extreme conditions found in vehicles. The paper discusses various techniques to improve object
detector model deployment efficiency, such as hardware-friendly pruning and quantization, matrix
multiplication operations reduction, and hardware and software co-design, by combining pruning,
quantization, knowledge distillation, etc. along with hardware optimization such as parallel factors
adjustment, resource allocation, etc .

2.2 Gene Lewis, Object Detection for Autonomous Vehicles

The document discusses the development of an object detection system for autonomous vehicles.
The goal is to create a system that can accurately detect objects in real-time without the need for
extensive preprocessing or deep evaluations. The proposed system, called SimpleNet, achieves a
prediction accuracy of approximately 12.83% on a validation set and can make predictions at a rate
of 11 frames per second. The document begins by explaining the importance of object detection in
autonomous driving systems and the challenges associated with it. It then provides an overview of

3
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 2. Literature Review
related work, including the OverFeat, VGG16, Fast R-CNN, and YOLO models, which have been
successful in object detection tasks but may not be suitable for autonomous driving due to their
complexity and computational requirements.
The proposed SimpleNet architecture is based on Convolutional Neural Networks (CNNs), which
have shown significant results in computer vision tasks. The architecture consists of multiple
layers followed by a final convolution/non-linearity and a dense fully-connected layer for prediction.
The document explains the choice of functions and activation functions used in SimpleNet. The
training process involves optimizing the network parameters using the Adaptive momentum (Adam)
algorithm. Two different loss functions, L1 and L2, are experimented with, and it is found that the
L2 loss function performs better in terms of overfitting and training convergence. Experiments are
conducted using the KITTI Object Detection Benchmark dataset, which includes large and high-
resolution images. The results show that the SimpleNet model achieves a mean Average Precision
(mAP) score of 12.83% on the validation set. However, the model’s prediction speed is slower
compared to other methods, such as the YOLO architecture.
Qualitative analysis of the model’s performance is also provided. Successful and unsuccessful
examples of detections are examined to understand the strengths and limitations of the model.
Suggestions for future work include improving the loss function to capture desired metrics more
accurately, increasing prediction speed by sharing computation, and using larger datasets for train-
ing and testing. Overall, the document presents the development and evaluation of SimpleNet, an
object detection system for autonomous vehicles. While the system shows reasonable prediction
accuracy and runs in near-real-time, there is room for improvement in terms of accuracy and speed.

2.3 Ross Greer, Akshay Gopalkrishnan, Nachiket Deo, Akshay

Rangesh, Mohan Trivedi, SALIENT SIGN DETECTION IN
SAFE AUTONOMOUS DRIVING: AI WHICH REASONS
OVER FULL VISUAL CONTEXT
This paper presents a new approach to detecting traffic signs that emphasizes their impact on a
driver’s decisions. The authors argue that traditional methods of traffic sign detection do not take
into account the importance of certain signs in influencing driving behavior. To address this issue,
they created the LAVA Salient Signs Dataset, which includes an annotated salience property for
each sign. They then used a custom salience loss function, called Salience-Sensitive Focal Loss, to
train a Deformable DETR object detection model. The results show that the model trained with
this loss function outperforms a model trained without it, with regards to recall of both salient
signs and all signs combined. The authors also discuss the potential applications of this research
in the field of autonomous driving. They argue that detecting and recognizing traffic signs is an
important module for an autonomous vehicle to observe and interact with its surroundings in a safe

B.Tech CS(AI) 2020-24 4

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 2. Literature Review
manner. The Safety of the Intended Functionality (SOTIF) process examines highly automated
systems for possible hazards and triggering events for unintended behaviors.
Failure to detect a sign crucial to driving performance would be considered a triggering event,
based on system limitations. Therefore, the authors suggest that their approach to salient sign
detection could improve the safety of autonomous vehicles by ensuring that important signs are
not missed. Overall, the paper presents a promising approach to improving traffic sign detection
in autonomous driving systems. By emphasizing the importance of certain signs, the authors have
shown that their method can significantly improve recall performance. The potential applications
of this research in the field of autonomous driving are also significant, as ensuring the safety of
these systems is of utmost importance.

2.4 Yizhe Wang , Xiaoguang Yang , Hailun Liang , and Yangdong

Liu,A Review of the Self-Adaptive Traffic Signal Control Sys-
tem Based on Future Traffic Environment
This paper provides an overview of the self-adaptive traffic signal control system, which is designed
to improve traffic efficiency on urban road networks. The system is based on a combination of real-
time traffic data collection, machine learning algorithms, and adaptive signal control strategies.
By continuously monitoring traffic flow and adjusting signal timing parameters in response to
changing demand, the system is able to reduce congestion, improve travel times, and enhance
overall safety on the road. The paper begins by discussing the current state of traffic signal control
systems, which are often based on outdated models and limited data sources. It then introduces
the concept of self-adaptive control, which uses real-time data to adjust signal timing parameters in
response to changing traffic conditions. The authors describe the key technologies that have enabled
the development of this system, including machine learning algorithms, wireless communication
networks, and advanced traffic sensors.
The paper then provides a detailed overview of the self-adaptive traffic signal control system,
including its architecture, data collection methods, and adaptive control strategies. The authors
describe how the system uses machine learning algorithms to analyze traffic data and predict
future demand, and how it adjusts signal timing parameters in real time to optimize traffic flow.
They also discuss the potential benefits of implementing this system in urban areas, including
reduced congestion, improved travel times, and enhanced safety. Finally, the paper concludes by
highlighting some of the challenges that need to be addressed in order to fully realize the potential
of the self-adaptive traffic signal control system. These include issues related to data privacy,
cybersecurity, and system interoperability. Overall, the paper provides a comprehensive overview
of this innovative traffic control system and its potential to transform urban transportation.

B.Tech CS(AI) 2020-24 5

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 2. Literature Review
2.5 Luca Mora , Xinyi Wu, Anastasia Panori,Mind the gap: De-
velopments in autonomous driving research and the sustain-
ability challenge
This paper provides a comprehensive overview of the developments in autonomous driving research
and the sustainability challenge. The file is structured into three main sections: an evidence-
based statement on the current state of AV research, a discussion of latent knowledge discovery in
AV research, and a presentation of the core thematic areas in AV research. The evidence-based
statement provides an overview of the current state of AV research, highlighting the progress that
has been made in recent years and the challenges that still need to be addressed. The statement
also discusses the limitations of the study and details its contribution to the field. The discussion of
latent knowledge discovery in AV research focuses on the multi-granulation approach to synthesizing
knowledge in the field.
The approach involves extracting the words and phrases of greatest significance, arranging these
text elements into groups of core topics, and facilitating the identification of the foremost thematic
area emerging from each group of topics. The discussion also highlights the challenges that scholars
face in keeping up with the evolving intellectual structure of the research area. The presentation of
the core thematic areas in AV research is based on a combination of citation-based community detec-
tion methods and topic modelling techniques. Thirteen core thematic areas are identified, including
safety and security, human factors, and environmental impact. The presentation also includes a
discussion of the implications of these thematic areas for the sustainability challenge. Overall, this
PDF file provides a valuable resource for anyone interested in the developments in autonomous driv-
ing research and the sustainability challenge. The evidence-based statement, discussion of latent
knowledge discovery, and presentation of core thematic areas provide a comprehensive overview of
the field and highlight the challenges that still need to be addressed.

B.Tech CS(AI) 2020-24 6

Chapter 3

Methodology

The methodology proposed in the paper is an Inductive Logic Programming (ILP)-based approach
for traffic sign detection in Autonomous Vehicles (AVs). The approach uses a set of logical rules
to detect traffic signs based on their high-level features, such as shape, colour, and text. The ILP
system is trained on a base dataset of traffic sign images without any adversarial perturbation.
Positive examples contain stop sign images, and negative examples include other traffic signs ex-
cluding stop sign images. The ILP system then induces a hypothesis (logical rule) based on the
logical facts and the names of the positive and negative examples. The ILP system recognizes new
traffic signs using this induced rule.

3.1 Inductive Logical Programming

3.1.1 Overview
Inductive Logic Programming (ILP) is a machine learning technique that uses logical rules to induce
hypotheses from a set of positive and negative examples. The goal of ILP is to learn a hypothesis
(rule) using a few positive and negative examples and Background Knowledge (BK); this induced
rule, alongside BK, should cover as many positive and as few negative examples as possible. ILP
is a logic-based AI approach that has the potential for human-like abstraction and reasoning. It
complements deep learning because logic programs are interpretable and data-efficient, leading
them towards a strong generalization. Moreover, these rule-based approaches, which are explicitly
symbolic, are sometimes considered safer than neural approaches. P uses a set of logical rules to
represent the knowledge about a domain. These rules are expressed in a formal language, such as
Prolog, and are used to induce new rules from a set of positive and negative examples. The ILP
system first generates a set of candidate rules based on the background knowledge and the positive
examples. It then evaluates each candidate rule based on its coverage of the positive and negative
examples. The ILP system selects the best rule that covers the most positive examples and the

7
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 3. Methodology
fewest negative examples .

Figure 3.1: ILP- based traffic sign classifier

3.1.2 General Architecture

1. Background Knowledge (BK): The BK contains the logical rules and facts that represent the
knowledge about the domain. The BK is used to guide the ILP system in inducing a hypothesis
(rule) that is consistent with the domain knowledge.
Positive and Negative Examples: The positive and negative examples are used to induce a
hypothesis (rule) that covers as many positive examples and as few negative examples as possible.
The positive examples contain instances of the target concept, while the negative examples contain
instances of other concepts that are not the target concept.
ILP System: The ILP system is responsible for inducing the hypothesis from the BK and the
examples. The ILP system generates a set of candidate rules based on the BK and the positive
examples and evaluates each candidate rule based on its coverage of the positive and negative
examples. The ILP system selects the best rule that covers the most positive examples and the
fewest negative examples. Hypothesis Evaluation: The induced rule, along with the BK, can be
used to recognize new examples in the domain. The ILP system evaluates the induced rule on a
test set of examples to measure its accuracy and generalization performance.
Incremental Learning: Using BK makes ILP incremental. For example, new rules and facts can
be added to the BK to improve the accuracy of the induced rule .
The data collection module is responsible for collecting the base data set and the adversarial
test data set. The base data set includes traffic sign images without any adversarial perturbation,

B.Tech CS(AI) 2020-24 8

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 3. Methodology

Figure 3.2: The general architecture flow

while the adversarial test data set includes stop signs perturbed by subtle, camouflage graffiti, and
camouflage art attacks viewed from different angles. The data collection module ensures that the
data is diverse and representative of the real-world scenarios.

3.2 Modules
3.2.1 Data Collection
The data collection module is responsible for collecting the base data set and the adversarial test
data set. The base data set includes traffic sign images without any adversarial perturbation,
while the adversarial test data set includes stop signs perturbed by subtle, camouflage graffiti, and
camouflage art attacks viewed from different angles. The data collection module ensures that the
data is diverse and representative of the real-world scenarios.

3.2.2 Feature Extraction

The feature extraction module is responsible for extracting the features of the traffic sign images.
The features are extracted using a combination of color and shape features. The color features
include the RGB values of the pixels, while the shape features include the number of edges and the
area of the sign. The feature extraction module ensures that the relevant information is extracted
from the images to enable accurate detection.

B.Tech CS(AI) 2020-24 9

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 3. Methodology

Figure 3.3: Base data set for training and testing

3.2.3 Logical Representation

The logical representation module is responsible for representing the features in a logical form using
predicates and logical operators. The logical representation includes the names of the positive and
negative examples and their corresponding logical facts. The logical representation module ensures
that the features are represented in a way that is amenable to logical reasoning.

3.2.4 Inductive Logic Programming

The Inductive Logic Programming (ILP) module is responsible for inducing a hypothesis (rule)
that covers as many positive examples and as few negative examples as possible. The ILP system
generates a set of candidate rules based on the BK and the positive examples and evaluates each
candidate rule based on its coverage of the positive and negative examples. The ILP system selects
the best rule that covers the most positive examples and the Hypothesis Evaluation e fewest negative
examples. The ILP module ensures that the detection system is accurate and generalizes well.

3.2.5 Hypothesis Evaluation

The hypothesis evaluation module is responsible for evaluating the induced rule on a test set
of examples to measure its accuracy and generalization performance. The induced rule, along
with the BK, can be used to recognize new examples in the domain. The hypothesis evaluation
module ensures that the detection system is evaluated rigorously and its performance is measured
accurately.

B.Tech CS(AI) 2020-24 10

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 3. Methodology
3.2.6 Adversarial Attack Detection
The adversarial attack detection module is responsible for evaluating the robustness of the ILP
stop sign detector against adversarial attacks. The RP 2 and AdvCam attack algorithms are used
to generate visual adversarial perturbations, such as black and white stickers attached to a traffic
sign to mislead the classifier. The adversarial attack detection module ensures that the detection
system is robust against adversarial attacks and can be trusted in real-world scenarios.

Figure 3.4: Base data set for training and testing

B.Tech CS(AI) 2020-24 11

Chapter 4

Result and Discussions

The study highlights the limitations of the current Deep Neural Network (DNN) based approaches
for traffic sign detection in AVs. DNN-based classifiers require a large amount of data for training,
are vulnerable to adversarial attacks or natural noise, and are not explainable. The proposed
ILP-based approach mimics humans in traffic sign detection and uses high-level features of a sign,
such as color and shape, for detection. Therefore, this method is data-efficient, explainable, and
able to withstand adversarial attacks that cannot easily deceive humans. The study shows that

Figure 4.1: Results

the proposed ILP-based approach outperforms the DNN-based approach in terms of accuracy and
robustness against adversarial attacks. The ILP-based approach achieved 100% accuracy on the
base data set and the RP 2 and AdvCam adversarial test data sets, while the DNN-based approach
achieved 100% accuracy on the base data set but only 66.6% accuracy on the AdvCam test data
set. The results indicate that the ILP-based approach with only a handful of training data can

12
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 4. Result and Discussions
induce logical rules easily understandable by humans. Furthermore, it significantly outperforms
the deep learning approach regarding adversarial attacks. .
The study also shows that the ILP-based approach is more interpretable and explainable than
the DNN-based approach. The induced rule can be easily understood and interpreted by humans,
making it easier to identify and correct any errors or biases in the system. The study concludes
that the ILP-based approach is a promising alternative to the current DNN-based approaches for
traffic sign detection in AVs. The ILP-based approach is more robust against adversarial attacks,
more interpretable and explainable, and can be easily updated and maintained. .
The proposed approach has various potential applications in the fields of Autonomous Vehicles,
Traffic Management, Road Safety, Security, Industrial Automation, Smart Cities, Emergency Ser-
vices, Environmental Monitoring, Agriculture, and Education. The more robust stop sign detection
system can contribute to improving safety, reliability, and efficiency in these domains. For instance,
the proposed approach can be integrated into AVs to improve their safety and reliability. The more
robust stop sign detection system can help AVs to navigate through complex traffic scenarios more
effectively. The proposed approach can also be used in traffic management systems to detect and
monitor traffic signs in real-time. The system can help traffic management authorities to monitor
traffic signs’ condition and detect any damage or vandalism. .
In conclusion, the study presents a novel approach for traffic sign detection in AVs that is
more robust, interpretable, and explainable than the current DNN-based approaches. The results
of the study show that the proposed approach outperforms the DNN-based approach in terms of
accuracy and robustness against adversarial attacks. The proposed approach has various potential
applications in different domains, and it can contribute to improving safety, reliability, and efficiency
in these domains. .

B.Tech CS(AI) 2020-24 13

Chapter 5

Applications

1. Autonomous Vehicles :
The primary application of this project is in the field of Autonomous Vehicles (AVs). The
ILP-based approach for stop sign detection can be integrated into AVs to improve their safety
and reliability. The more robust stop sign detection system can help AVs to navigate through
complex traffic scenarios more effectively.

2. Traffic Management :
The project’s approach can also be used in traffic management systems to detect and monitor
traffic signs in real-time. The system can help traffic management authorities to monitor
traffic signs’ condition and detect any damage or vandalism. .

3. Road Safety :
The project’s approach can contribute to improving road safety by detecting stop signs more
accurately and reliably. This can help reduce the number of accidents caused by drivers failing
to stop at stop signs.

4. Security:
The project’s approach can also be used in security systems to detect and monitor traffic signs
in sensitive areas such as airports, military bases, and government buildings. The system can
help detect any unauthorized access or suspicious activity in these areas.

5. Industrial Automation :
The project’s approach can be used in industrial automation systems to detect and monitor
safety signs in factories and warehouses. The system can help ensure that workers follow
safety protocols and prevent accidents in the workplace.

6. Environmental Monitoring :
The project’s approach can be used in environmental monitoring systems to detect and mon-

14
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 5. Applications
itor signs related to environmental hazards such as radiation, chemical spills, and natural
disasters. The system can help detect and respond to environmental hazards more effectively.

7. Agriculture :
The project’s approach can be used in agriculture to detect and monitor signs related to crop
health, irrigation, and pest control. The system can help farmers optimize crop yields and
reduce the use of pesticides and water.

8. Emergency Services :
The project’s approach can be used by emergency services such as ambulances and fire trucks
to detect and navigate through traffic signs more effectively. This can help reduce response
times and improve emergency services’ efficiency.

9. Smart Cities :
The project’s approach can be used in smart city systems to detect and monitor traffic signs
in real-time. The system can help optimize traffic flow and reduce congestion in urban areas.

10. Education :
The project’s approach can be used in educational settings to teach students about traffic
signs and their meanings. The system can help students learn about traffic safety and improve
their understanding of traffic signs.

B.Tech CS(AI) 2020-24 15

Chapter 6

Conclusion

The study highlights the limitations of the current Deep Neural Network (DNN) based approaches
for traffic sign detection in AVs. DNN-based classifiers require a large amount of data for training,
are vulnerable to adversarial attacks or natural noise, and are not explainable. The proposed ILP-
based approach mimics humans in traffic sign detection and uses high-level features of a sign, such
as color and shape, for detection. Therefore, this method is data-efficient, explainable, and able to
withstand adversarial attacks that cannot easily deceive humans. The results of the study show
that the proposed ILP-based approach outperforms the DNN-based approach in terms of accuracy
and robustness against adversarial attacks. The ILP-based approach achieved 100% accuracy on
the base data set and the RP 2 and AdvCam adversarial test data sets, while the DNN-based
approach achieved 100% accuracy on the base data set but only 66.6% accuracy on the AdvCam
test data set. The results indicate that the ILP-based approach with only a handful of training data
can induce logical rules easily understandable by humans. Furthermore, it significantly outperforms
the deep learning approach regarding adversarial attacks. The study also shows that the ILP-based
approach is more interpretable and explainable than the DNN-based approach. The induced rule
can be easily understood and interpreted by humans, making it easier to identify and correct any
errors or biases in the system. The study concludes that the ILP-based approach is a promising
alternative to the current DNN-based approaches for traffic sign detection in AVs. The ILP-based
approach is more robust against adversarial attacks, more interpretable and explainable, and can
be easily updated and maintained.
The proposed approach has various potential applications in the fields of Autonomous Vehicles,
Traffic Management, Road Safety, Security, Industrial Automation, Smart Cities, Emergency Ser-
vices, Environmental Monitoring, Agriculture, and Education. The more robust stop sign detection
system can contribute to improving safety, reliability, and efficiency in these domains. For instance,
the proposed approach can be integrated into AVs to improve their safety and reliability. The more
robust stop sign detection system can help AVs to navigate through complex traffic scenarios more
effectively. The proposed approach can also be used in traffic management systems to detect and

16
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach 6. Conclusion
monitor traffic signs in real-time. The system can help traffic management authorities to monitor
traffic signs’ condition and detect any damage or vandalism.
In conclusion, the study presents a novel approach for traffic sign detection in AVs that is
more robust, interpretable, and explainable than the current DNN-based approaches. The results
of the study show that the proposed approach outperforms the DNN-based approach in terms of
accuracy and robustness against adversarial attacks. The proposed approach has various potential
applications in different domains, and it can contribute to improving safety, reliability, and efficiency
in these domains. The proposed approach can be a significant step towards achieving safe and
trustworthy autonomous driving.

B.Tech CS(AI) 2020-24 17

Chapter 7

Bibliography

1. Abhishek Balasubramaniam and Sudeep Pasricha,Object Detection in Autonomous Vehicles:

Status and Open Challenges,Colorado State University 2022 - arxiv.org

2. Gene Lewis,Object Detection for Autonomous Vehicles,Stanford University Stanford, Stan-

ford University, Stanford, CA, 2018

3. Ross Greer; Akshay Gopalkrishnan; Nachiket Deo;Akshay Rangesh,Autonomous Driving: AI

which reasons over Full Visual Context,2023,arxiv.org

4. Yizhe Wang , Xiaoguang Yang , Hailun Liang , and Yangdong Liu,Signal Control System
Based on Future TrafficEnvironment 2018,hindawi.com

5. Luca Mora, Xinyi Wu ,Anastasia Panori,Mind the gap: Developments in autonomous driving
research and the sustainability challenge
2020 - Elsevier

18
Appendix A

Presentation Slides

CONTENTS
EXPLAINABLE AND TRUSTWORTHY TRAFFIC SIGN DETECTION FOR SAFE
AUTONOMOUS DRIVING: AN INDUCTIVE LOGIC PROGRAMMING  Introduction

APPROACH  Literature Review

 DNN
 Limitations of DNN
 Adversarial Attacks on AI Perception
GUIDED BY PRESENTED BY
 ILP Architecture
DR. PRIYA C V ANANDHU E S
 Methodology
ASST. PROF, MITS (MUT20CA012)
 Performance
 Advantages
 Applications
 Conclusion
1 2
 References

(a) Slide 1 (b) Slide 2

INTRODUCTION LITERATURE REVIEW

Details of Paper Methodology Limitations

 Traffic sign detection is a critical component of autonomous driving systems.

Object Detection in Autonomous - Object detection involves localization and - Two-stage detectors have higher accuracy
Vehicles: Status and Open Challenges classification of objects in images or videos. but are slower compared to single-stage
 Current deep neural network (DNN) based classifiers have shown to be vulnerable to adversarial attacks, Abhishek Balasubramaniam and Sudeep Pasricha - Two-stage detectors use region proposals detectors.
are not explainable, and require a large amount of data for training. Colorado State University and object classification, while single-stage - They can struggle with detecting objects
2022 - arxiv.org detectors perform both tasks in a single under challenging conditions such as low
 This paper proposes an Inductive Logic Programming (ILP) based approach for traffic sign detection in stage. lighting, or complex backgrounds.
autonomous vehicles. - Object detectors may have difficulties
with detecting small or heavily distorted
 The approach mimics human perception and uses high-level features of a sign, such as its shape, colour, objects accurately.
and text, for detection.
Object Detection for Autonomous - A network called SimpleNet achieves a - Lower Accuracy: The document
 This method is data-efficient, explainable, and able to withstand adversarial attacks that cannot easily Vehicles mean Average Precision (mAP) accuracy of acknowledges that SimpleNet achieves a
deceive humans. Gene Lewis approximately 12.83% on the validation relatively lower mean Average Precision
Stanford University set. (mAP) accuracy compared to many high-
Stanford, CA - It highlights the importance of accurate quality object detectors.
[email protected] object detection for autonomous - Model Size: SimpleNet consists of 12
3 4
Stanford University, Stanford, CA, 2018 navigation and identifies unique challenges layers, which could be considered relatively
in the autonomous driving setting. large in terms of the model's depth.

(c) Slide 3 (d) Slide 4

19
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach A. Presentation Slides

Details of Paper Methodology Limitations

Salient Sign Detection in Safe Salience refers to the potential of a sign to - It does not provide detailed information DNN
Autonomous Driving: AI which directly influence the driver's immediate about the specific performance metrics and
reasons over Full Visual Context decision. It propose a traffic sign detection evaluation results.
Ross Greer; Akshay Gopalkrishnan; Nachiket model that emphasizes performance on - The limitations of the LAVA Salient Signs
Deo;Akshay Rangesh salient signs. They create the LAVA Salient Dataset, such as potential biases or
2023,arxiv.org Signs Dataset, which includes annotations limitations in sign diversity,
for sign salience. are not addressed.  DNNs compute their internal parameters in forward pass and then iteratively refine them during
A Review of the Self-Adaptive Traffic - The self-adaptive traffic signal control - The existing self-adaptive traffic signal backpropagation to effectively extract input data features. These advantages give DNNs a greater learning
Signal Control system is an effective measure for relieving control systems have not fully utilized the capability, outperforming other methods in tasks such as computer vision, natural language processing, machine
System Based on Future Traffic urban traffic congestion. potential of abundant real-time traffic data. translation, speech recognition, genomics, quantitative trading, and self-driving cars.
- It adjusts signal timing parameters in real- - The theory, methods, and techniques of
Environment  DNN is conventionally used for traffic sign detection in AVs and many papers have been published based on the
Yizhe Wang , Xiaoguang Yang , Hailun Liang , and
time based on seasonal changes and short- these systems have lagged the progress of
Yangdong Liu, 2018,hindawi.com
term fluctuations in traffic demand. key basic technologies. same. However, DNN poses many challenges to safe prediction and only provides an accuracy of 66.6%. These
limitations are overcome by using Inductive Logic Programming, thus obtaining an accuracy of 100%.
Mind the gap: Developments in - The paper addresses the expanding - The analysis is based on data from 50
autonomous driving research and the scientific knowledge on autonomous- years of AV research, which may not
sustainability challenge driving technology and the challenge of capture the most recent developments in
Luca Mora, Xinyi Wu ,Anastasia Panori information overload for researchers. the field.
2020 - Elsevier - It proposes a multi-granulation approach - The paper does not explore the reasons 5 6
to latent knowledge discovery and behind the lack of attention to
synthesis in large-scale research domains. sustainability implications in AV research.

(e) Slide 5 (f) Slide 6

LIMITATIONS OF DNN ADVERSARIAL ATTACKS ON AI PERCEPTION

Deep Neural Networks (DNNs) play a significant role in developing perception systems for AVs. However, DNNs face This section survey a sample of successful adversarial attacks in autonomous driving that easily deceived DNN-based
significant challenges that must be addressed before AVs can be deployed safely. The major challenges facing DNN- vision classifiers. An adversarial attack aims to generate adversarial examples as the input for machine learning
based vision systems in autonomous driving are discussed below.
systems.
 DNN-based systems are often considered "black boxes" because their logic is not transparent. Since it is difficult
to explain how the system makes the prediction, it is challenging to debug them when they make a wrong  Robust Physical Perturbations (RP2) technique fools Convolutional Neural Network (CNN) based road sign
decision. classifier in the physical world under various distances and viewpoints using different robust visual adversarial
perturbations. This approach caused targeted misclassification, which changed a stop sign into a speed limit sign
 DNNs face significant challenges when it comes to learning from small data and achieving out-of-distribution for the AI system. They also proposed a disappearance attack, causing a stop sign hidden from state-of-art object
generalizability and transferability to new domains. In real-world security domains, there is often a lack of large,
detectors like Mask R-CNN and YOLO.
annotated, and carefully curated data sets to train these systems. This can make it difficult for DNNs to acquire
knowledge from a few examples and transfer it to new domains.  An Adversarial Camouflage (AdvCam) approach generated adversarial photos to fool a DNN classifier at various
 DNNs are vulnerable to adversarial attacks and can be deceived easily. In adversarial cases, minor perturbations detecting angles and distances. With a few stains invisible to humans, this technique can cause the classifier to
will lead to misclassifications with high confidence. misclassify the objects, such as misidentifying a stop sign as a "barber shop" with .82% confidence.

7 8

(g) Slide 7 (h) Slide 8

METHODOLOGY

 The base data set is utilised for training the ILP systems (Aleph and Metagol).
 First, we randomly select an equal number of positive and negative examples in each run, so the default accuracy
is 50% for this training data set.
 Next, the ILP-based systems try to find a hypothesis that covers as many positive and as few negative examples as
possible.
 Then the remaining examples in the data set are used as a test data set for evaluation to determine the accuracy.
 This process is repeated one hundred times, and average accuracy is calculated for each certain number of
positive and negative examples in the training set.

Targeted physical perturbation by a) AdvCam and b) RP_2 misleading DNN classifiers, SL45 9 10

is speed limit 45 sign.

Fig(1)

(i) Slide 9 (j) Slide 10

B.Tech CS(AI) 2020-24 20

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach A. Presentation Slides

INDUCTIVE LOGIC PROGRAMMING ILP ARCHITECTURE

 The first step is pre-processing all the images, including training and test images, and turning them into a
 Inductive Logic Programming (ILP) is a machine learning method which uses logic-based representation and symbolic representation to provide BK. In the pre-processing phase, high-level features of traffic sign images,
inference. Depending on the type of logical inference and the search algorithm, there are different ILP systems, including colour, shape, text and digits, are extracted and represented as a set of logical facts for the next step.
such as Aleph and Metagol. For feature extraction, computer vision tools such as OpenCV can extract high-level features using low-level
features such as pixel colours or colour gradients.
 Due to a logic-based representation and inference, ILP has the potential for human-like abstraction and
reasoning. These logic-based AI approaches can learn unknown complex tasks with only a few examples.  In the next step, a set of positive and negative training examples (E) and a set of logical facts as BK extracted from
the previous step will be provided to the ILP system. The system aims to learn a hypothesis H such that B,H |= E
 ILP aims to learn a hypothesis (rule) using a few positive and negative examples and Background Knowledge (BK); where |= is logical entailment.
this induced rule, alongside BK, should cover as many positive and as few negative examples as possible. For
inducing the rules, BK should include all essential predicates to represent the relevant information.  Metagol is employed in the other experiment. It is an ILP system based on Meta Interpretive Learning (MIL)
implemented in Prolog. By instantiating metarules, MIL learns logic programs from examples and BK. In addition,
 One of the advantages of ILP is its ability to use BK, including facts and rules in the form of logical expressions, MIL not only learns the recursive definition and fetches higher-order meta-rules but also supports predicate
which could be related. In ILP, choosing appropriate BK based on well-selected features is essential to obtaining invention.
good results. Moreover, using BK makes ILP incremental.
11 12

(k) Slide 11 (l) Slide 12

ILP ARCHITECTURE ILP ARCHITECTURE

 One of the experiments uses Aleph5; it is an old ILP system developed in Prolog. Aleph’s algorithm resolves the
relationship between the determination predicate and the determining predicate to generate a general theory.

 To further explain the process of converting images into a set of logical facts, one positive example is taken, a
stop sign named p1, and one negative example, a speed limit sign named n1. In the preprocessing stage, the
high-level features of these traffic signs were extracted to be included in the BK. The details of these features and
their corresponding logical representation are presented.
 These logical facts, together with the names of the positive and negative examples (such as sign(p1) as a positive
and sign(n1) as a negative example), will enable the ILP system to induce a hypothesis (logical rule). Finally, the
ILP system recognizes the new traffic signs using this induced rule.
13 14

Fig(2)

(m) Slide 13 (n) Slide 14

PERFORMANCE

 Metagol can find a hypothesis with 100% accuracy on the test data set including only one positive and one
negative example. In comparison, Aleph starts learning with at least two positive and two negative examples with
around 65% accuracy. Aleph can reach the same level of accuracy as Metagol by learning from eight positive and
negative examples. According to these results, Metagal is more data-efficient than Aleph.
 The hypothesis (a logic program) induced by Metagol with only one set of positive and negative examples is the
same as the learned rule by Aleph with eight positive and negative examples. It is entirely accurate on the base
test data set and is shown below:
 Alogrithm:
t r a f f i c _ s i g n (A, s t o p _ s i g n ) : −
has_word (A,A_w1 ) ,
c l o s e l y _ma t c h (A_w1 , s t o p )

15 16

Base data set for training and testing

Fig(3)

(o) Slide 15 (p) Slide 16

B.Tech CS(AI) 2020-24 21

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach A. Presentation Slides

PERFORMANCE ADVANTAGES

 This learned rule is completely explainable and matches human interpretation. The rule says the traffic sign "A" is
a stop sign when the two literals has_word(A, A_w1) and closely_match(A_w1, stop) hold, i.e. if the sign contains  Using high-level features of a sign, such as its shape, colour, and text, for traffic sign detection in AVs has
a word and that word closely matches stop, that sign would be predicted a stop sign. several advantages.

 The performance of this hypothesis is evaluated on the base data set and attack data sets, including RP_2 (subtle,  It makes the system more robust against adversarial attacks, as it mimics human-like perception and is
camouflage graffiti and camouflage art attacks) and AdvCam with different stains. The accuracy of this rule on all less susceptible to the limitations of current DNN classifiers.
test data sets is 100%.  It is more explainable, making it possible to debug AVs. Thirdly, it requires minimal training data and is
 The DNN-based classifier is trained on the GTSRB data set, which contains more than 50,000 images. The Aleph- efficient.
based classifier is trained on eight positive and negative examples, while the Metagol approach is trained on only  The proposed ILP-based approach is able to detect categories of traffic signs, making it possible to have
one negative and one positive example. It shows that while ILP-based systems can learn from small amounts of real-time interaction with drivers towards customized autonomous driving.
data, they are more resilient to noise and adversarial attacks.

17 18

(q) Slide 17 (r) Slide 18

APPLICATIONS CONCLUSION

 This paper proposed an ILP-based approach for traffic sign detection in autonomous vehicles.

 This approach is more robust against adversarial attacks, more explainable, and requires minimal
 The main application of this work is to improve the safety and reliability of AVs by providing a more
training data.
trustworthy and explainable traffic sign detection system.
 The paper mainly focuses on the application of the proposed ILP-based approach for stop sign detection  The proposed technique mimics human perception and uses high-level features of a sign, such as its
in AVs. However, the approach can be extended to detect other types of traffic signs as well. shape, colour, and text, for detection.
 Additionally, the concept of using high-level features of a sign, such as its shape, colour, and text, can be  The results shows that this approach is able to detect categories of traffic signs and is more accurate
applied to other computer vision tasks beyond traffic sign detection. For example, it can be used for than DNN-based classifiers on adversarial examples.
object recognition, face recognition, and other image classification tasks.
 This proposed technique has the potential to improve the safety and reliability of autonomous driving
systems.

19 20

(s) Slide 19 (t) Slide 20

REFERENCES

 https://www.sciencedirect.com/topics/computer-science/inductive-logic-programming
 Walker T, O'Reilly C, Kunapuli G, Natarajan S, Maclin R, Page D and Shavlik J. Automating the ILP setup task. Proceedings
of the 20th international conference on Inductive logic programming. (253-268)
 Balasubramaniam, A., and S. Pasricha. "Object Detection in Autonomous Vehicles: Status and Open Challenges. arXiv
2022." arXiv preprint arXiv:2201.07706.
 Mind the gap: Developments in autonomous driving research and the sustainability challenge Luca Mora, Xinyi Wu ,
Anastasia Panori,2020 – Elsevier
 Object Detection in Autonomous Vehicles: Status and Open Challenges Abhishek Balasubramaniam and Sudeep Pasricha
Colorado State University 2022 - arxiv.org
 A Review of the Self-Adaptive Traffic Signal Control System Based on Future Traffic Environment Yizhe Wang , Xiaoguang
Yang , Hailun Liang , and Yangdong Liu, 2018,hindawi.com
 Salient Sign Detection in Safe Autonomous Driving: AI which reasons over Full Visual Context Ross Greer; Akshay
Gopalkrishnan; Nachiket Deo;Akshay Rangesh 2023,arxiv.org
21

(u) Slide 21

B.Tech CS(AI) 2020-24 22

Appendix B

Sample Reference Papers

Object Detection for Autonomous Vehicles

Gene Lewis
Stanford University
Stanford, CA
[email protected]

Abstract fast generally precludes the use of image preprocess-

ing that often boosts detection performance, such as
In this work, we have examine an approach to deep ob- extracting SIFT descriptors, region proposals, or slid-
ject detection that makes bounding box predictions for an ing windows. Thus, a detector for an autonomous car
image without the need for expensive preprocessing or ex- must remain reasonably accurate while operating on
pensive deep evaluations; the resulting DIY network, Sim- strictly the input image.
pleNet, gives reasonable prediction accuracy and runs in
• Car computing systems are often heavily memory con-
near-real-time. SimpleNet is capable of making predic-
strained, and so it is often infeasible to store and run
tions at a rate of approximately 11 frames per second, and
detectors with large amounts of parameters, especially
achieves a mAP accuracy of around 12.83% on our valida-
with large input image volumes. In particular, this con-
tion set.
strains the depth of neural network approaches.

1. Introduction 2. Related Work

2.1. OverFeat
In recent years, there has been a significant increase
in research interest supporting the development of the au- OverFeat [10] is a Convolutional Neural Network model
tonomous vehicle, which is an automobile platform capable released in 2013 that jointly performs object recognition,
of sensing and reacting to it’s immediate environment in an detection, and localization. OverFeat is one of the most
attempt to navigate roadways without human intervention. successful detection models to date, winning the localiza-
The task of environment sensing is known as perception, tion task in the ImageNet Large Scale Visual Recognition
and often consists of a number of subtasks such as object Challenge 2013. OverFeat is eight layers deep, and depends
classification, detection, 3D position estimation, and simul- heavily on an overlapping scheme that produces detection
taneous localization and mapping (SLAM)[6]. boxes at multiple scales and iteratively aggregates them to-
In many autonomous driving systems, the object detec- gether into high-confidence predictions.
tion subtask is itself one of the most important prerequisites
to autonomous navigation, as this task is what allows the car
2.2. VGG16
controller to account for obstacles when considering possi- VGG16 [11] was an attempt to usurp OverFeat’s dom-
ble future trajectories; it therefore follows that we desire inance in object classification and detection by exploring
object detection algorithms that are as accurate as possible. the effects of extreme layer depth. A 16-layer and 19-layer
Many high-quality object detectors have seen astounding model was produced, setting new benchmarks on localiza-
improvements in recent years [5]; however, the object de- tion and classification in the ImageNet ILSVRC 2014 Chal-
tection problem ported to the autonomous driving setting lenge.
presents unique challenges not addressed by many leading
techniques. These challenges include: 2.3. Fast R-CNN

• Car controllers often must solve optimization prob- Fast R-CNN [3] is a model that attempts to capture the
lems at least once a second to achieve feasible control; accuracy of deeper models while improving their speed.
this means that the car controller must receive detec- Fast R-CNN predicts on region proposals, and utilizes
tion results in a similar timeframe. shared computation per region proposal and truncated SVD
factorizations to speed up the training and prediction time
• The requirement that the entire detection pipeline be of the model.

(a) Reference paper 1

23
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach B. Sample Reference Papers

Hindawi
Journal of Advanced Transportation
Volume 2018, Article ID 1096123, 12 pages
https://doi.org/10.1155/2018/1096123

Review Article
A Review of the Self-Adaptive Traffic Signal Control
System Based on Future Traffic Environment

Yizhe Wang , Xiaoguang Yang , Hailun Liang , and Yangdong Liu

Key Laboratory of Road and Traffic Engineering of the Ministry of Education, Tongji University, 4800 Cao’an Road,
Shanghai 201804, China

Correspondence should be addressed to Xiaoguang Yang; [email protected]

Received 9 January 2018; Revised 20 April 2018; Accepted 23 April 2018; Published 27 June 2018

Academic Editor: Ludovic Leclercq

Copyright © 2018 Yizhe Wang et al. This is an open access article distributed under the Creative Commons Attribution License,
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The self-adaptive traffic signal control system serves as an effective measure for relieving urban traffic congestion. The system
is capable of adjusting the signal timing parameters in real time according to the seasonal changes and short-term fluctuation
of traffic demand, resulting in improvement of the efficiency of traffic operation on urban road networks. The development
of information technologies on computing science, autonomous driving, vehicle-to-vehicle, and mobile Internet has created a
sufficient abundance of acquisition means for traffic data. Great improvements for data acquisition include the increase of available
amount of holographic data, available data types, and accuracy. The article investigates the development of commonly used self-
adaptive signal control systems in the world, their technical characteristics, the current research status of self-adaptive control
methods, and the signal control methods for heterogeneous traffic flow composed of connected vehicles and autonomous vehicles.
Finally, the article concluded that signal control based on multiagent reinforcement learning is a kind of closed-loop feedback
adaptive control method, which outperforms many counterparts in terms of real-time characteristic, accuracy, and self-learning
and therefore will be an important research focus of control method in future due to the property of “model-free” and “self-
learning” that well accommodates the abundance of traffic information data. Besides, it will also provide an entry point and technical
support for the development of Vehicle-to-X systems, Internet of vehicles, and autonomous driving industries. Therefore, the related
achievements of the adaptive control system for the future traffic environment have extremely broad application prospects.

1. Introduction technology, computer technology, and system science. The

self-adaptive control system can adjust the signal timing
The amount of motor vehicles and correspondent travel de- parameters in real time according to the control target of the
mand are continuously increasing with economic and social manager (such as the minimum delay of the intersection) and
development. The frequent occurrence of traffic congestion the arrival characteristics of the traffic flow at the intersection.
in urban road network has negative impacts on economy and Compared with timing control and actuated control, the self-
environment. Due to the limited land resources of large cities adaptive control system can make better use of the overall
and restrictions to transportation infrastructure construction traffic capacity of the road network and effectively improve
from socioeconomic factors, to apply traffic management the efficiency of road network traffic.
and control measures in a reasonable and effective way, The traffic data collected by the current traffic control
improve the efficiency of existing transportation facilities, system using induction loop detector and other existing
and accommodate the growing traffic demand in big cities sensors is limited. With the advancement of the wireless
have become significant research contents for counteracting communication technologies and the development of the
urban traffic congestion. vehicle-to-vehicle (V2V) and vehicle to infrastructure (V2I)
Traffic control is one of the most important technical systems, called Connected Vehicle or V2X, there is an oppor-
means to regulate traffic flow, improve the congestion, and tunity to optimize the operation of urban traffic network
even reduce emissions. Its progress and development has by cooperation between traffic signal control and driving
always been accompanied by the development of information behaviors. This dissertation proposed a series of cooperative

(a) Reference paper 2

B.Tech CS(AI) 2020-24 24

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach B. Sample Reference Papers

Journal of Cleaner Production 275 (2020) 124087

Contents lists available at ScienceDirect

Journal of Cleaner Production

journal homepage: www.elsevier.com/locate/jclepro

Review

Mind the gap: Developments in autonomous driving research and the

sustainability challenge
Luca Mora a, *, Xinyi Wu b, Anastasia Panori c
a
The Business School, Edinburgh Napier University, Edinburgh, EH14 1DJ, United Kingdom
b
School of Social and Political Science, The University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom
c
White Research, Brussels, 1060, Belgium

a r t i c l e i n f o
Scientific knowledge on autonomous-driving technology is expanding at a faster-than-ever pace. As a
Article history: result, the likelihood of incurring information overload is particularly notable for researchers, who can
Received 19 April 2020 struggle to overcome the gap between information processing requirements and information processing
Received in revised form capacity. We address this issue by adopting a multi-granulation approach to latent knowledge discovery
25 August 2020 and synthesis in large-scale research domains. The proposed methodology combines citation-based
Accepted 31 August 2020 community detection methods and topic modelling techniques to give a concise but comprehensive
Available online 11 September 2020 overview of how the autonomous vehicle (AV) research field is conceptually structured. Thirteen core
Handling editor. Prof. Jiri Jaromir Kleme
s thematic areas are extracted and presented by mining the large data-rich environments resulting from
50 years of AV research. The analysis demonstrates that this research field is strongly oriented towards
examining the technological developments needed to enable the widespread rollout of AVs, whereas it
Keywords:
Autonomous vehicle largely overlooks the wide-ranging sustainability implications of this sociotechnical transition. On ac-
Research developments count of these findings, we call for a broader engagement of AV researchers with the sustainability
Text mining concept and we invite them to increase their commitment to conducting systematic investigations into
Topic modelling the sustainability of AV deployment. Sustainability research is urgently required to produce an evidence-
Sustainability based understanding of what new sociotechnical arrangements are needed to ensure that the systemic
Knowledge gap technological change introduced by AV-based transport systems can fulfill societal functions while
meeting the urgent need for more sustainable transport solutions.
© 2020 Elsevier Ltd. All rights reserved.

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Latent knowledge discovery in AV research: a multi-granulation perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Data and methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.1. Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2. Topic modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.3. Post-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.1. CL.01: The Urban Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.2. CL.02: Real-time motion planning of multi-AV operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.3. CL.03: Multi-sensors and fusion systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.4. CL.04: Road boundaries and extended curbs detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.5. CL.05: Motion planning for agricultural machinery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.6. CL.06: Lane detection and connected technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.7. CL.07: Motion planning for underwater intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.8. CL.08: Obstacle detection and avoidance in different conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.9. CL.09: Traffic sign recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

* Corresponding author.
E-mail addresses: [email protected], [email protected] (L. Mora).

https://doi.org/10.1016/j.jclepro.2020.124087
0959-6526/© 2020 Elsevier Ltd. All rights reserved.

(a) Reference paper 3

B.Tech CS(AI) 2020-24 25

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach B. Sample Reference Papers

Object Detection in Autonomous Vehicles:

Status and Open Challenges
Abhishek Balasubramaniam and Sudeep Pasricha
Colorado State University

Abstract— Object detection is a computer vision task that has Table 1: SAE J3016 levels of automation
become an integral part of many consumer applications today SAE Level Name Driving Environment
such as surveillance and security systems, mobile text recognition, Monitor
and diagnosing diseases from MRI/CT scans. Object detection is 0 No Automation
also one of the critical components to support autonomous driving. 1 Driver Assistance Human Driver
Autonomous vehicles rely on the perception of their surroundings 2 Partial Driving Automation
to ensure safe and robust driving performance. This perception 3 Conditional Driving Automation
system uses object detection algorithms to accurately determine 4 High Driving Automation ADAS System
objects such as pedestrians, vehicles, traffic signs, and barriers in 5 Full Driving Automation
the vehicle's vicinity. Deep learning-based object detectors play a
vital role in finding and localizing these objects in real-time. This Tesla was the first company to commercialize AVs with
article discusses the state-of-the-art in object detectors and open their Autopilot system in 2014 that offered level 2 autonomy
challenges for their integration into autonomous vehicles. [6]. Tesla AVs were able to travel from New York to San
Francisco in 2015 by covering 99% of the distance
autonomously. In 2017, Volvo launched their Drive Me feature
I. INTRODUCTION with level 2 autonomy, with their vehicles traveling
Autonomous vehicles (AVs) have received immense autonomously around the city of Gothenburg in Sweden under
attention in recent years, in large part due to their potential to specific weather conditions [7]. Waymo has been testing its
improve driving comfort and reduce injuries from vehicle AVs since 2009 and has completed 200 million miles of AV
crashes. It has been reported that more than 36,000 people died testing. They also launched their driverless taxi service with
in 2019 due to fatal accidents on U.S. roadways [1]. AVs can level 4 autonomy in 2018 in the metro Phoenix area in USA
eliminate human error and distracted driving that is responsible with 1000 – 2000 riders per week, among which 5 – 10% of the
for 94% of these accidents [2]. By using sensors such as rides were fully autonomous without any drivers [8]. Cruise
cameras, lidars, and radars to perceive their surroundings, AVs Automation started testing a fleet of 30 vehicles in San
can detect objects in their vicinity and make real-time decisions Francisco with level 4 autonomy in 2017, launched their self-
to avoid collisions and ensure safe driving behavior. driving Robotaxi service in 2021 [9]. Even though Waymo and
AVs are generally categorized into six levels by the SAE Cruise support level 5 autonomy, their AVs are classified as
J3016 standard [3] based on their extent of supported level 4 because there is still no guarantee that they can operate
automation (see Table 1). While level 0 – 2 vehicles provide safely in all weather and environmental conditions.
increasingly sophisticated support for steering and acceleration, AVs rely heavily on sensors such as cameras, lidars, and
they heavily rely on the human driver to make decisions. Level radars for autonomous navigation and decision making. For
3 vehicles are equipped with Advanced Driver Assistance example, Tesla AVs rely on camera data with six forward
Systems (ADAS) to operate the vehicle in various conditions, facing cameras and ultrasonic sensors. In contrast, Cruise AVs
but human intervention may be requested to safely steer, brake, use a sensor cluster that consists of a radar in the front while
or accelerate as needed. Level 4 vehicles are capable of full self- camera and lidar sensors are mounted on the top of the AV to
driving mode in specific conditions but will not operate if these provide a 360-degree view of the vehicle surroundings [9]. One
conditions are not met. Level 5 vehicles can drive without of the main tasks involved in achieving robust environmental
human interaction under all conditions. perception in AVs is to detect objects in the AV vicinity using
Automotive manufactures have been experimenting with software-based object detection algorithms. Object detection is
AVs since the 1920s. The first modern AV was designed as part a computer vision task that is critical for recognizing and
of CMU NavLab’s autonomous land vehicle project in 1984 localizing objects such as pedestrians, traffic lights/signs, other
with level 1 autonomy that was able to steer the vehicle while vehicles, and barriers in the AV vicinity. It is the foundation for
the acceleration was controlled by a human driver [4]. This was high-level tasks during AV operation, such as object tracking,
followed by an AV designed by Mercedes-Benz in 1987 with event detection, motion control, and path planning.
level 2 autonomy that was able to control steering and The modern evolution of object detectors began 20 years
acceleration with limited human supervision [5]. Subsequently, ago with the Viola Jones detector [10] used for human face
most major auto manufacturers such as General Motors, Bosch, detection in real-time. A few years later, Histogram of Oriented
Nissan, and Audi started to work on AVs. Gradient (HOG) [11] detectors became popular for pedestrian

(a) Reference paper 4

B.Tech CS(AI) 2020-24 26

Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach B. Sample Reference Papers

International Conference on The Enhanced Safety of Vehicles (ESV), Japan, April 2023

SALIENT SIGN DETECTION IN SAFE AUTONOMOUS DRIVING: AI WHICH REASONS OVER FULL
VISUAL CONTEXT

Ross Greer
Akshay Gopalkrishnan
Nachiket Deo
Akshay Rangesh
Mohan Trivedi
Laboratory for Intelligent & Safe Automobiles 1
University of California San Diego
USA

Paper Number 23-0333

ABSTRACT

Detecting road traffic signs and accurately determining how they can affect the driver’s future actions is a critical
task for safe autonomous driving systems. However, various traffic signs in a driving scene have an unequal impact
on the driver’s decisions, making detecting the salient traffic signs a more important task. Our research addresses
this issue, constructing a traffic sign detection model which emphasizes performance on salient signs, or signs that
influence the decisions of a driver. We define a traffic sign salience property and use it to construct the LAVA Salient
Signs Dataset, the first traffic sign dataset that includes an annotated salience property. Next, we use a custom
salience loss function, Salience-Sensitive Focal Loss, to train a Deformable DETR object detection model in order
to emphasize stronger performance on salient signs. Results show that a model trained with Salience-Sensitive Focal
Loss outperforms a model trained without, with regards to recall of both salient signs and all signs combined.
Further, the performance margin on salient signs compared to all signs is largest for the model trained with
Salience-Sensitive Focal Loss.

INTRODUCTION

Detecting and recognizing traffic signs is an important module for an autonomous vehicle to observe and interact
with its surroundings in a safe manner. The Safety of the Intended Functionality (SOTIF) process [1] examines
highly automated systems for possible hazards and triggering events for unintended behaviors; in this framework,
failure to detect a sign crucial to driving performance would be considered a triggering event, independent of the
hazardous events, based on system limitations. Accordingly, detection systems are continuously improved to push
the safe limits of their operation. Until recently, standard object detectors operated by proposing regions of interest
or considering a standard set of anchors or window centers within an image, and classifying the contents of the
found region. These approaches are typically limited by the span of the convolutional filters which drive them; these
filters operate on local windows, or with a pre-determined span and spacing. While the reach of the convolutional
filters can be tuned to spread and cover the entire image, doing so creates massive computational costs or creates
gaps in coverage. As a solution, the popular transformer model has been proposed as a means of reasoning over the
entire image and bringing forward features relevant to the region of interest. To minimize computational costs, this
approach has been further refined to include a stage of learning (via a limited number of deformable attention heads)
where an image should be sampled to extract meaningful relational features to the region of interest. This approach
is known as the Deformable Detection Transformer, introduced in technical detail in the following section.

1
cvrr.ucsd.edu

Greer 1

(a) Reference paper 5

B.Tech CS(AI) 2020-24 27

Appendix C

Viva questions and answers

1.What is the proposed approach for traffic sign detection in AVs?

The proposed approach is an Inductive Logic Programming (ILP) based approach for traffic
sign detection in AVs.

2. What are the limitations of the current Deep Neural Network (DNN) based ap-
proaches for traffic sign detection in AVs?
The limitations of the current DNN-based approaches for traffic sign detection in AVs are that
they require a large amount of data for training, are vulnerable to adversarial attacks or natural
noise, and are not explainable.

3. How does the proposed ILP-based approach differ from the DNN-based approach?
The proposed ILP-based approach mimics humans in traffic sign detection and uses high-level
features of a sign, such as color and shape, for detection. Therefore, this method is data-efficient,
explainable, and able to withstand adversarial attacks that cannot easily deceive humans.

4. What is the accuracy of the proposed ILP-based approach on the base data set and
the RP 2 and AdvCam adversarial test data sets?
The proposed ILP-based approach achieved 100% accuracy on the base data set and the RP 2
and AdvCam adversarial test data sets

5. What are the potential applications of the proposed approach?

The proposed approach has various potential applications in the fields of Autonomous Vehicles,
Traffic Management, Road Safety, Security, Industrial Automation, Smart Cities, Emergency Ser-
vices, Environmental Monitoring, Agriculture, and Education.

28
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic
Programming Approach C. Viva questions and answers
6. What is the significance of the proposed approach towards achieving safe and
trustworthy autonomous driving?
The proposed approach can be a significant step towards achieving safe and trustworthy au-
tonomous driving by providing a more robust, interpretable, and explainable stop sign detection
system.

7. How can the proposed approach contribute to improving safety and reliability
in AVs?
The more robust stop sign detection system can help AVs to navigate through complex traffic
scenarios more effectively, contributing to improving safety and reliability in AVs.

8. What is the accuracy of the DNN-based approach on the AdvCam test data set?
The DNN-based approach achieved only 66.6% accuracy on the AdvCam test data set.

9. The DNN-based approach achieved only 66.6% accuracy on the AdvCam test
data set.
Some potential challenges that the proposed ILP-based approach may face in real-world sce-
narios include variations in lighting conditions, weather conditions, and the presence of occlusions
or other objects that may interfere with the detection of traffic signs.

10. How can the proposed approach be updated and maintained over time?
The proposed approach can be updated and maintained over time by incorporating new data
and rules into the system as they become available. This can help to improve the accuracy and
robustness of the system over time.

B.Tech CS(AI) 2020-24 29