2020 International Conference on Artificial Intelligence and Signal Processing (AISP)

Deep Learning Techniques for Obstacle Detection

and Avoidance in Driverless Cars
Nischal Sanil, Pasumarthy Ankith Naga venkat, Rakesh V, Rishab Mallapur, Mohammed Riyaz Ahmed
School of Electronics and Communication Engineering
REVA University Bangalore India-64
email: [email protected], [email protected], [email protected], [email protected], [email protected]

Abstract—With the advent of Internet of Things (IoT), The

realization of smart city seems to be very imminent. One of
the key parts of a cyber physical system of urban life is
transportation. This mission-critical application has attracted
many researchers in both academia and industry to investigate
driverless cars. In the domain of autonomous vehicles, intelligent
video analytics is very critical. By the advent of deep learning
many neural networks based learning approaches are under
consideration. This work tries to implement obstacle detection
and avoidance in a self-driven car. One of advanced neural
network called Convolutional Neural Network (CNN) is exploited
for real time video/image analysis using an IOT device. This
project makes use of a raspberry pi which is responsible for
controlling the car and performing inference using CNN, based
on its current input. The model trained has achieved an accuracy
of 88.6% and are in good consent with expected performance.
Index Terms—Internet of Things, Autonomous car, Deep
Learning, Convolutional Neural Network, Raspberry PI.
Fig. 1. Autonomous car environment.
I. I NTRODUCTION
Great men said that it is better to prevent and prepare than situation, the struggle for a parking spot is resolved and as it
to repent and repair. The current automobile safety techniques could park on its own and pick up the owner where ever he
like ABS and airbags have reached a plateau. The technology is. It caters to the transportation needs of people of all ages
is progressing and so is the mindset, the aim is not to survive [3].
a car accident but is to prevent it from happening. the world All self-driving cars are the same provided they are on the
has come down such that we believe in the machines we make same level. Driverless cars are classified on the basis of their
more than ourselves and this machine can prevent millions of ability to function automatically and are categorized under
deaths due to accidents and it would be called as the self- different levels. There are 5 different levels, starting from
driving car [1]. LEVEL1 where the driver has to have his hands-on, Which
A self-driving car or an autonomous car / driverless car means that driver assistance is mandatory. In LEVEL2, the
by definition is a vehicle that is capable of sensing its driver can have his Hands off as it is a form of partial automa-
environment and moving with little or no human input [2]. tion where the technology can simultaneously control steering
In this monocular vision based (single input) car, will get its and speed at the same time, without driver intervention for
input from the pi camera. Raspberry Pi has been used as a short periods. LEVEL3 form of automation enables the driver
processing unit in this project, which takes the input through to have his Eyes off as the vehicle is capable of taking full
the raspberry pi camera and drives the car by sending control control and operating during select parts of a journey when
signals to the L293d driver module, which in turn controls certain operating conditions are met. The LEVEL4 or the high
the car. The core of the project is the Convolutional Neural automation technology doesnt need a human driver meaning
Network which maps the raw image pixels into features that that he can take his Mind off. The vehicle can essentially do
detect objects and navigates the car. all the driving, but the driver can intervene and take control
The self-driving car is one of the best inventions made by as needed. Finally, the last level being LEVEL5 or the full
mankind and would relieve many problems such as driving automation is a technology which can take care of the driving
when you are sleepy or inebriated. it would improve safety scenario in all extreme conditions and the humans are just
during the night drives which is a necessary need in the current passengers [4].

978-1-7281-4458-0/20/$31.00
Authorized ©2020
licensed use limited to: Indian Institute IEEE Hyderabad. Downloaded on October 08,2022 at 14:02:30 UTC from IEEE Xplore. Restrictions apply.
of Technology
 2020 International Conference on Artificial Intelligence and Signal Processing (AISP)

Fig. 2. Evolution of the self driving car.

For the beauty of the rose, we also water the thorns, TABLE I
The main drawback of the vision-based method is that it D RIVERLESS CARS BY VARIOUS O RGANIZATIONS
is not robust to illumination changes. A trained model with Organization Year Disengagement Miles per Dis- Levels
added weights is fed into the raspberry pi which is used per 1000 miles engagement
BMW 2005 219.51 4.6 2
for navigating a car through the obstacles. Due to the lack Waymo 2009 0.09 11154.3 4
of computation power in raspberry pi, techniques such as Audi 2010 * * 3
reinforcement learning (in which the output depends on the GM 2011 0.19 5204.9 2
Nissan 2013 4.75 210.5 2+
state of the current input and the next input depends on the Tesla 2014 Not tested Not tested 2
output of the previous input) is not viable [5]. Zoox 2014 0.52 1,922.8 3
MercedesBenz 2015 682.52 1.5 4
Nuro 2016 0.97 1028.3 5
In this paper we have implemented a obstacle detection and pony.ai 2016 0.98 1022.3 *
avoidance driverless car using Convolution neural networks. In Uber 2016 2608.46 0.4 2+
section II we have surveyed on evolution of driverless car with Apple 2017 871.65 1.1 *
practical innovation, Section III shows the implementation of ∗ Data unavailable.
Details are arranged in chronological order.
hardware and the model architecture. Section IV discuss the
results of the model.
Road and transport research laboratory from the United King-
II. L ITERATURE R EVIEW
dom experimented an autonomous car, Citroen DS in 1966
Though self-driving cars are moving to be a prominent which communicated with the magnetic cables installed on
technology of the century, the efforts on designing the model the roadside [7]. Citroen DS showed exceptional results as it
started way back in the 1920s. Houdina Radio Control Co.[4], was capable of traveling 130 km/hr in any weather conditions,
a radio equipment firm founded by Francis P Houdin exhibited but the installation cost was still too expensive [8].
the first radio control car in 1925. This car made use of a In 1979, Hans Moravec [9] a Ph.D. student built a cart
receiving antenna on its tonneau which was followed by a which used vision inputs for decision making. This was the
transmitter from a certain distance. The radio signals drove first driving project that made use of image processing. This
small motors that directed every movement of the car. This, cart successfully managed to navigate a cluttered room with no
however, still requires an individual to constantly monitor the human intervention. Later, over the years advanced versions of
car as a driver would, thereby not satisfying the contemporary the cart were introduced. One such work was the CMU rover
requirements [6]. which was more capable and explored new directions. The use
A similar attempt of a radio-controlled car was designed in of image processing reduced the cost compared to the Citroen
1939, with an embedded circuitry on the roadside, thereby DS and other similar proposals, but the complexity increased
expelling the need for an individual to constantly monitor as maintaining the hardware was a demanding task and the
the status of the car but increases the expenses of the road. cart was very slow. Yann Lecun et. al [11], introduced the

Authorized licensed use limited to: Indian Institute of Technology Hyderabad. Downloaded on October 08,2022 at 14:02:30 UTC from IEEE Xplore. Restrictions apply.
 2020 International Conference on Artificial Intelligence and Signal Processing (AISP)

Fig. 3. Semantic diagram of the autonomous car.

convolutional neural network for the first time in 1998 which through Wi-fi in devices such as laptop and smartphones. Next
was used for handwriting recognition. This led to a remarkable is to build a track so that we can train the car. As we run the
leap in deep learning as a neural network model was capable car on the track it collects the image data through the raspberry
of identifying various patterns of human handwritings in an pi camera and the input command (forward, stop, right, left).
image automatically with the help of training examples rather This data will be used to train the neural network model.
than using a traditionally handcrafted feature extraction fol-
lowed by a classifier. The advancement in technology and the C. Convolution neural network:
reduction of cost in computation over the years made CNN’s
extremely popular. Various improvements and variants of the Convolutional neural networks(CNN) have become ubiqui-
model were introduced due to its favorable results. Mariusz tous in computer vision. CNN is most popular because of
Bojarski et al [10], from the NVIDIA Corporation, introduced reliable results on object recognition and detection that are
a fully autonomous car in 2016. The car had three cameras(left, useful in real-world applications. The typical use of convo-
right and center) mounted behind the windshield constantly lutional networks is on classification tasks, where the output
capturing the inputs at 10 FPS, furthermore increase in the FPS to an image is a single class label that needs to be classified.
would not provide much information. The car made use of the Convolution is a function derived from two given functions by
outputs CNN model to handle the steering. Data acquisition integration which expresses how the shape of one is modified
was conducted in various lighting and weather conditions, by the other. Unlike neural networks, the input to CNN is an
making the learning more effective. The trained model was image. The convolution layer comprises of a set of independent
loaded on an NVIDIA DRIVE PX and tested on roads with filters. The filter is slid over the complete image and the dot
30 FPS, to handle abrupt variations in decision making [12]. product is taken between the filter and chunks of the input
Many leading companies have shown their interests in similar image. Each filter is independently convolved with the image
projects like Google, Tesla, etc. and end up with feature maps. There are several uses that we
gain from deriving a feature map, reducing the size of the
III. E XPERIMENTAL SET- UP image by preserving it’s semantic information is one of them.
A. Hardware and Software
D. Neural Network Architecture:
The car framework was built using a stripped-down RC
car, we kept the main frame and the two DC motors of the The images are provided as an input to the model. Using
RC car. An L293d motor driver was used to control the two OpenCV the images of size (mention dimension) is fed to the
DC motors. L293d contains two full H-bridges which enable model. The First layer of the model is a convolution layer
us to control two DC motors bi-directionally, one for the consists of 32 filters of size/dimension 3x3, followed by a
acceleration and another for steering movement of the car. dropout layer of 20%, the dropout will avoid the overfitting
All the connections were done as mentioned. The Raspberry of the model. Then another convolution layer of 64 filters
Pi is a small but full-featured computer on a single board, with 3x3 dimension. A Max pooling of 2x2 is applied, as
this is a practical, portable and inexpensive device. We used it reduces the dimensionality and allows assumptions to be
Raspberry pi 3B for processing as well as for controlling the made about features contained in the sub-regions binned. After
car. A raspberry camera is attached at the front of the car; it max pooling, a dropout of 20% is applied followed by a
will collect the image data during the training process. flattening layer which will convert a 3D image to a 1d array.
The dense layer of 128 neurons with a ReLu activation is
B. Data collection: applied. Finally, the output (dense) layer consists of 4 outputs
For Building a Neural network model we need lots of data. with Softmax activation. We are estimating the losses using
The data is collected during the training process. Initially, the categorical cross-entropy and we used Adam optimizer of the
car needs to be controlled wirelessly. To do so we used VNC model since it is the best optimizer that provides reliable
viewer which enables us to control Raspberry Pi wirelessly results.

Authorized licensed use limited to: Indian Institute of Technology Hyderabad. Downloaded on October 08,2022 at 14:02:30 UTC from IEEE Xplore. Restrictions apply.
 2020 International Conference on Artificial Intelligence and Signal Processing (AISP)

need. In this paper, we have presented a monocular vision

based self-driving car, in which an image input is taken
through the camera and a CNN model is used to take a decision
accordingly. For a driverless car, Accuracy and efficiency are
directly proportionate to the amount of training data and the
quality of the object detection model. This concept calls for
an advanced and safe future for all citizens in the field of
transportation due to its ability to relay passengers or goods
without human intervention.
ACKNOWLEDGMENT
The authors would like to thank all the fellow researchers of
PSYSQUARE (an OPC) for their valuable inputs and enduring
support. The authors would also like to express their deep
gratitude to REVA University for their unconditional support
in carrying this work.
R EFERENCES
[1] Wei, Junqing, et al. ”Towards a viable autonomous driving research
platform.” 2013 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2013.
Fig. 4. Convolutional model architecture [2] Greenblatt, Nathan A. ”Self-driving cars and the law.” IEEE spectrum
53.2 (2016): 46-51.
[3] Do, Truong-Dong, et al. ”Real-Time Self-Driving Car Navigation Using
Deep Neural Network.” 2018 4th International Conference on Green
Technology and Sustainable Development (GTSD). IEEE, 2018.
[4] Howard, Daniel, and Danielle Dai. ”Public perceptions of self-driving
cars: The case of Berkeley, California.” Transportation Research Board
93rd Annual Meeting. Vol. 14. No. 4502. 2014.
[5] Urmson, Chris. ”Self-driving cars and the urban challenge.” IEEE
Intelligent Systems 23.2 (2008): 66-68.
[6] Bimbraw, Keshav. ”Autonomous cars: Past, present and future a review
of the developments in the last century, the present scenario and
the expected future of autonomous vehicle technology.” 2015 12th
International Conference on Informatics in Control, Automation and
Robotics (ICINCO). Vol. 1. IEEE, 2015.
[7] Treml, Michael, et al. ”Speeding up semantic segmentation for au-
tonomous driving.” MLITS, NIPS Workshop. Vol. 1. 2016.
[8] Levinson, Jesse, et al. ”Towards fully autonomous driving: Systems and
Fig. 5. Working model of autonomous car algorithms.” 2011 IEEE Intelligent Vehicles Symposium (IV). IEEE,
2011.
[9] Moravec, Hans P. ”The stanford cart and the CMU rover.” Autonomous
Robot Vehicles. Springer, New York, NY, 1990. 407-419., Hans P.
IV. R ESULTS ”The stanford cart and the CMU rover.” Autonomous Robot Vehicles.
Springer, New York, NY, 1990. 407-419.
A miniature version of a self-driving car has been imple- [10] Bojarski, Mariusz, et al. ”End to end learning for self-driving cars.”
mented, which is capable of decision making with the help arXiv preprint arXiv:1604.07316 (2016).
[11] LeCun, Yann, and Yoshua Bengio. ”Convolutional networks for images,
of Convolutional Neural Networks. The car was trained on speech, and time series.” The handbook of brain theory and neural
a self made track which was around 12 to 14 meters long. networks 3361.10 (1995): 1995.
During the training phase, the car had achieved an accuracy [12] Hee Lee, Gim, Friedrich Faundorfer, and Marc Pollefeys. ”Motion
estimation for self-driving cars with a generalized camera.” Proceedings
of 88.6%, which can be improved further by hyperparameter of the IEEE Conference on Computer Vision and Pattern Recognition.
tunning and more data collection. The car functioned very well 2013.
for most for its journey but had difficulties when taking sharp
turns, which was due to the limitation of the motor’s turning
capabilities. The car had successfully mimicked the driving
patterns within the scope of the dataset and would require
more data accumulation for adjusting to new scenarios.

V. C ONCLUSION
Artificial intelligence is one technology that would revo-
lutionize the world as we know it and lead us head first
into the future. One of the applications of AI and a tranche
of Computer vision is the autonomous self-driving car. This
robotic car might be today’s luxury but is surely tomorrow’s

Authorized licensed use limited to: Indian Institute of Technology Hyderabad. Downloaded on October 08,2022 at 14:02:30 UTC from IEEE Xplore. Restrictions apply.