SSRN Id4296815
SSRN Id4296815
SSRN Id4296815
Abstract:
In today's world where all tasks can be done with the speed of a single click, autonomous vehicles have come into
picture which not only ensures safe self- driving but also an effortless driving experience. The demand of such
vehicle tends to grow much more in the near future due to which technologies supporting these vehicles are being
improvised and researched. Computer Vision enables the computer to see, understand and make observations just
as Artificial Intelligence helps the computer to think. Computer Vision is a branch of AI which helps the computer
systems to visualize and obtain information through various inputs just as the human eye works in the human
body. Computers can gain immense and accurate information through digital images, visuals as well as videos
when trained. Computer vision trains algorithms and machines so that they can make self -driving easier. It uses
sensor technology to recognize objects and people on the lane. Computer vision algorithms when trained can target
the safety of the passengers as well as hassle free driving. The motive of this paper is to introduce the research
trends as well as technologies being used for autonomous vehicles using computer vision; we also aim to discuss
challenges, advantages as well as future scope of these vehicles.
© 2022 – Authors.
1. Introduction
The earliest cars were introduced in 1700 which used electricity and steam; later in the next century gas
powered cars were first introduced. The evolution of vehicles has been since ancient times introducing new
and interesting features and now we have driver less cars which are a boon to humanity as it takes away all the
effort by the driver to reach his destinations without having to sit at one place and concentrate. The first idea
of driver less vehicles was introduced back in 1939. This concept has reached the next level where they are
now known as autonomous vehicles. An autonomous vehicle uses a combination of sensors, Li-DAR, Radar
artificial intelligence, radars, and cameras to operate, without any human interference. These sensors observe
and inspect the area around the vehicle before it takes any critical decision related to the action to be taken
place. After that a localization map is created which can be 2D or 3D. The objective of the map is to recognize
the fixed objects around the vehicle. The dynamic objects are removed by removing all Li-DAR points that
are found in the area encircling detected dynamic objects. Currently computer vision uses expensive cameras
to avoid inconsistency and produce more precise results. Autonomous vehicles are still in an early stage where
along with benefits there are certain drawbacks as well; hence these vehicles are sometimes incapable of
driving through city traffic and bumpy roads. Research is still being made to improve the expectancy of these
vehicles and computer vision technology is highly used in these areas.
There are a number of ways in which computer vision can be used in an autonomous vehicle, some of them
can be:
● Detecting and classifying objects using Li-DAR sensors and cameras. By classifying static and
dynamic objects the vehicle can decide to move accurately and quickly. It will help avoid collisions
and mishaps.
● Creation of 3D maps can help the vehicles to know their surroundings and to recognize obstacles in
the lane and to avoid them. This can also help in prediction of accidents and hence ensure the safety
of the passengers.
● Collecting large data for training algorithms using cameras and sensors can help the vehicle to predict
more accurately and easily the types of actions that are needed to be taken. This data can include
anything like traffic signals, potholes, maintenance of roads, etc.
Although the advancement in this field there is a lot to be discovered and improvised hence there are
challenges that these vehicles face. Collecting the data for training is one of them. Vehicles without human
interference need a lot of training and a variety of deep testing. Instead of using a data set it can be better if
real time images are captured while driving and are later used for training purposes. After collecting, labeling
of data requires a huge amount of manual work. The labeled data must be precise and accurate to help the
algorithm train and produce efficient results. Object detection is yet another challenge as the images need
classification and localization. AI in self-driving cars is an unending ocean full of discoveries and rapidly
advancing technological turns. Hence, we aim to dive more into these advantages and drawbacks in the paper
mainly related to the use of computer vision.
The remaining paper is set as follows, in Section II the working of Autonomous vehicles is elaborated in
brief, Section III presents the literature survey Next, in Section IV the companies which use Computer vision
in their autonomous vehicles are listed. The areas of application of computer vision are discussed in section V,
and then in section VI discusses the analysis of literature survey, section VII discusses the challenges faced
and possible solutions are presented by us. Then the final Section VII discusses the future scope and
conclusion.
Self-driving also called automated vehicles say that the vehicle is capable of making its own choices and
decisions. Autonomous vehicles depend on a lot of complex algorithms, sensors, maps, AI and machine
learning to derive its desired objective. The main motive of combining Computer Vision with Artificial
intelligence is to form a visual representation that can perform the job that humans do without humans while
making the right decisions. Autonomous vehicles use technology to replace drivers with automatic safety
features to traverse roads. Even though the idea of an effortless driving seems comforting to the audience there
are a lot of challenges that these vehicles are yet to overcome and hence a completely autonomous vehicle is
yet to run on the roads. Tesla being a famous example of a self-driving car, still needs active driver
supervision and hence does not make the vehicle completely autonomous. Autonomous cars need the driver to
give a destination and the car automatically finds the best route. Li-DAR sensors are then used to measure the
exact distance of an object and create a 3D map surrounding the car and the car's position is then checked
corresponding to the map. It's mounted on the car's top roof in a 360-degree rotating cylindrical compartment.
Li-DAR comprises a transmitter, a reflector, and a receiver. To image objects, it uses laser, ultra ultraviolet,
visible light, or infrared light. The emitter shoots a laser beam, which rebounds off a mirror that rotates at 10
revolutions per minute with the cylindrical housing. Cameras are used as sensors to give the best sensor an
accurate visual depiction of an autonomous car's nearby area. The cameras are fixed on every of the four sides
to give a 360° view. Ultra wide lenses are used in these vehicles to capture a panoramic view that helps the
driver with parking. Radars are also used in these vehicles which work on radio waves. They work at different
frequencies correlated to short range radars and medium range radars.
Radar is used majorly because it works with millimeter waves and offers precision. It also works better
than other sensors even under low range of vision. The Frequency Modulated Continuous Wave radar offers
high resolution and fast pulse emission, hence they are used in autonomous vehicles. The next technology that
is used when autonomous vehicles are concerned is ultrasonic sensors. These sensors work by measuring the
reflected signals. A transducer in ultrasonic sensors produces vibrations that emit ultrasonic pulses. The range
is determined by the frequency of the vibration; they are used widely in parking systems to detect the hurdles
behind the vehicle to avoid accidents.
2.1 Tables:
5. 2019 This research employs a semantic scene They have used an This algorithm On drivers, human
segmentation method to classify each unlabeled data set. showed better results positional
pixel according to the object class to in tasks of bounding prediction may be
which it belongs. box localization as conducted, which
K-Mean clustering and Expectation well as semantic now has
segmentation. applications in both
Maximization are two algorithms that
have been used to deal with picture semi-autonomous
segmentation issues such as pixel vehicles and
ADAS.
clustering.
6. 2018 This paper uses computer vision and They have used an The algorithm of the Sensors and high
deep learning to train the data. They opacity train data set model was tested on end tools are not
have also used neural network to for lane detection campus as a test using used hence this
classify traffic signs and vehicle consisting of 39209 a four wheeled remains as a model
detection. Binocular camera and GPU is images and test data electric car with a for understanding
used to understand the environment set consisting of 12630 binocular camera. The safety in self-
images. method can be used in driving but cannot
assisted driving be implemented.
research and is helpful
and reliable for safety
of self- driving.
7 2018 SSD7 detects autos and walkers with The data set for this Even though the This approach can
opacity, a data set for self-driving cars, paper was a Unity approach's processing be used in a more
in this paper. As shown with the engine simulation time is longer than sophisticated
simulation results, SSD7 was chosen environment. People, segmentation and environment to
because the network size is small and vehicles, streets, recognition alone, it train our network
the throughput is satisfactory. vegetation, and the air still achieves 30 from end to end,
were all present in the frames a second. leading in a more
virtual environment. powerful network.
We will now discuss the findings and research methods applied in various papers and articles which discuss
the application of computer vision in autonomous vehicles.
B. Object Detection :
Earlier methods have 2 ways to improvise the task performance: (i) creating a more efficient method
for withdrawing regions; (ii) improvising the classifying model. Procedures that utilize one deep
learning network have been proposed, involving the YOLO algorithm.
E. CNN classification :
Convolution Neural Network also said as COVNET. It is used to determine visual imagery. These
are proficient in identifying objects from pictures or video. All the different layers of CNN have
convolution filters for reducing dimensions. Computer vision works on images and image
segmentation is a very important step. There are 2 types of segmentation: Semantic segmentation
which categorizes image pixels into n number of classes. Instance segmentation instances of each
object are identified.
F. Current Situation :
In autonomous vehicles, computer vision is used to determine various objects. Today most computer
vision algorithms get competition from Li-DAR because Li-DAR is able to solve most of the
problems which cannot be solved by computer vision algorithms. Computer vision makes
autonomous driving safe for people.
G. Data Simulation :
TE Connectivity gives a powerful simulation tool for better speed data connectivity to let a big range
of professionals cross the limits of what is possible. Initiate your own creativity in a modern and
structured environment. Dive deep right in with our step-by-step method and start creating now.
Computer vision utilizes real-time images to develop a 3D map. With the use of 3D maps,
autonomous vehicles can decode the driving space for risk-free driving and also opt for alternate
routes in criteria of projected collision.
The next research conducted on this topic, focuses on the collaboration with the mono camera and a
previously trained model along with multi- regression technique. This previously trained model is re-trained
on a local data set with more images taken from a mono camera to detect cars from the road in the working
environment. Then the distance of the cars is calculated by depth equation. The research discusses an Anti-
collision algorithm which has various sections such as object detection in which a multi task algorithm is used
to obtain information using YOLO which is an object detection tool. The next factor is depth estimation using
regression techniques followed by lane and tracking algorithms separately. The lane assignment algorithm
uses geometry principles to help the cars stay in their lanes which are determined by each vehicle's
boundaries.
When it comes to safety, recognizing and regulating the environment is widely debated. Some
research findings used a binocular image sensor and the NVIDIA Jet son TX2 computing systems with E-net
and Open CV for traffic signal recognition, and surrounding vehicles used the MPC technique to make a
decision and choose on driving commands in the simulation model and for control in a real-world scenario.
Using dual-motor electric vehicles with this method at low speeds aids environmental perception and
accordance with the specified. After converting the image into gray scale Gaussian blur is applied to the
image followed by canny edge detection and Hough transformation. Road segmentation is done mainly using
CNN classification but the drawback that follows is the down sampling which results in loss of edge
information. Target object detection and traffic segmentation is done to find out the position and by MPC
decision making cars are able to drive slowly through the area.
Approximating the depth is considered a key element in providing the safety of a car and its
passengers. Various tools are available which consist of camera radar and Li-DAR which are enabled with
multiple camera vision. Estimating the space between camera lenses and the appropriate destination of an
object on images taken is necessary for building a stereo vision system. The system uses the concept of
triangulation to make space and depth decisions.
B. Detection of Accidents :
Autonomous cars can detect and report accidents in a number of ways, including sending alerts to
nearby support centers. Automobile safety is an important aspect in reducing accidents. Computer vision,
sensor fusion, localization, optimum path, and control unit are the five critical elements of self-driving cars. A
camera is being used to visualize the environment of the vehicle in computer vision. As a consequence, the
computer can retrieve specific details and characteristics from these photos. A solution to locate vehicles
which uses Gaussian Mixture Model; has been introduced by Hui et al, which tracks vehicles by a mean shift
algorithm, but the challenge with this solution is that it fails to perform well under harsh weather and traffic
conditions because the parameters are few in number for the solution to perform. Road vehicle accidents have
risen in the past few years and hence detecting accidents using computer vision is like a boon, but it also
comes with a lot of challenges.
5. Comparative Analysis
The Le-Net -5 architecture outperforms over all other methodologies. The Le-Net -5 model reduces
the training time drastically. The Le-Net -5 architecture outperforms over semantic scene segmentation, SSD7
methodology and other techniques. Table I gives the detailed comparison of various state of the arts
methodologies.
Along with all the advantages that autonomous vehicles come with, we cannot overlook the
challenges faced by them. Computer vision has made some amazing advancements in the past five years,
driven strongly by the increase in performance of parallel computing infrastructure. Especially on roads with
Indian traffic it is extremely challenging to let the car take on a complete autonomous mode. The major
reasons that these vehicles can face challenge are:
● Lack of proper infrastructure.
● Lack of road network planning.
● Issues with barriers and road blockers.
● Potholes and bad road conditions during rains.
● Allowance of pedestrians, vehicles as well as animals along with non-motorized vehicles
altogether on roads.
The vehicles also need extremely intelligent systems to adapt to the quick changing inputs such as
the ability to apply brakes, accelerate and steer. A User Interface, Path Optimization Module, Environment
Observation and Modeling Module, and Command and Control Module all have a role in the study and
implementation of autonomous car movement at a top standard. Forward collision warning, lane departure
and lane keeping, Pedestrian and object sensing, Traffic sign recognition, Headlight control, Landscape
mapping, Blind Spot Monitoring, and Vehicle to Vehicle Communication are all abilities on another level.
One of the biggest concerns is, the sensors ability to navigate in extreme weather, snow blizzards (white-
outs), heavy rain, and dust storms and if they can with current sensor technology overcome all these
conditions autonomously.
Another fact that has to be accepted is that even though self-driving cars might be better than manual
in majority cases but they don't go without accidents. Taking into consideration Tesla's Full Self- Driving,
which is the only company that has been close to giving a complete solution. The Tesla comes in two
categories - Using the main release software, they are really excellent at self-driving on freeway, but cannot
do more than assist the driver in “in-town” areas.
Using the “beta release” software, which is limited to drivers who volunteer for the program and
whom Tesla have vetted as having excellent driving skills. The beta software is still making occasional
mistakes - around one significant error per hour, hence cannot be completely dependent. The current
computer vision technology that is used in self-driving vehicles is endangered to attacks, where hackers
exploit the AI’s inputs so that it makes errors. For example, researchers have shown they can manipulate a
self-driving car from recognizing stop signs by sticking black and white labels on them.
Now let's discuss probable solutions:
Autonomous cars have come a long way since they were first introduced. In this paper we have
aimed to discuss how computer vision helps and affects the working of autonomous vehicles as well as the
challenges and future scope of this technology. Autonomous vehicles still need to adapt to a lot more
situations and the amount of testing and training the data has to be significantly large to avoid errors.
Presently, autonomous vehicles require certain human contact and hence are not completely adaptable to
changes in the environment, but it is expected that in the coming years, cars will become fully autonomous.
Computer vision study has the potential to make meaningful contributions toward the eventual aim of a fully
automated vehicle.
The future of autonomous vehicles will be completely connected to humans and objects. The future
self-driving cars may also be able to communicate with other similar cars and exchange information.
Additionally services such as in-built car Wi-Fi, cloud services, entertainment services can be incorporated to
make the passengers feel extreme comfort and luxury. Plenty of technological innovations will take place as
the world is introduced to newer technologies. Self-driving cars may also be electric, so they can ensure safety
of the environment along with the comfort of autonomy. There are a number of challenges while this comes
into actual picture, but the change has already started to take place and we can be sure of looking at such types
of fully autonomous cars in the coming years.
The upcoming autonomous cars may also lift people from their stops and transfer them to their
desired places on its own. Since there will be no interference of humans the traffic issues can also disperse as
the vehicles may have smart road traffic detection and avoidance sensors. Driving licenses also will not be
required hence the hassle of driving tests and other documentation will be no longer needed for the drivers.
The Internet of Things and communication technology along with Computer Vision techniques will be more
advanced in the upcoming years along with new pattern detection, algorithms and sensors as well as efficient
collision and object detection algorithms and methods to make the driving experience smoother to its
passengers.
References
[1] Dhananjai Chand,, Savyasachi Gupta, Ilaiah Kavati, Department of Computer Science and Engineering National
[2] Institute of Technology Warangal, India-506004 Computer Vision based Accident Detection for Autonomous Vehicles.
[3] Nitin Kanagaraj1, David Hicks1, Ayush Goyal1, Sanju Tiwari2 , Ghanapriya Singh3 Deep learning using computer vision
in self-driving cars for lane and traffic sign detection
[4] Raia Hadsell1 Pierre Sermanet1,2 Marco Scoffier1,2 Koray Kavukcuoglu1 (1) Courant Institute of Mathematical Sciences
New York University New York, NY USA Jan Ben2 Ayse Erkan1 Urs Muller2 Yann LeCun1
[5] (2) Net-Scale Technologies Morganville, NJ USA,Learning Long-Range Vision for Autonomous Off-Road Driving 2020
Intermountain Engineering, Technology and Computing (IETC)
[6] Self-Driving Cars: A Platform for Learning and Research.“Intel® realsenseTM lidar camera l515.” 2020 (Accessed 26
February 2020). [Online]. Available: https://www.intelrealsense.com/lidar-camera- l515.
[7] Patil, M.N., Brijesh Iyer, Rajeev Arya (2016). Performance Evaluation of PCA and ICA Algorithm for Facial Expression
Recognition Application. Advances in Intelligent Systems and Computing, vol 436. Springer, Singapore.
https://doi.org/10.1007/978-981-10-0448-3_81
[8] P. Deshpande and B. Iyer, "Research directions in the Internet of Every Things(IoET)," 2017 International Conference on
Computing, Communication and Automation (ICCCA), 2017, pp. 1353-1357, doi: 10.1109/CCAA.2017.8230008.
[9] Prachi Deshpande, S. C. Sharma, Sateesh K. Peddoju, Ajith Abraham, "Efficient Multimedia Data Storage In Cloud
Environment",Vol 39, No 4,431-442,2015.
[10] Prachi Deshpande, Sharma, S.C., Peddoju, S.K. (2016). Data Storage Security in Cloud Paradigm. Advances in Intelligent
Systems and Computing, vol 436. Springer, Singapore. https://doi.org/10.1007/978-981-10-0448-3_20
[11] S. Kamthekar and B.Iyer,"Cognitive Analytics for Rapid Stress Relief in Humans Using EEG Based Analysis of Tratak
Sadhana (Meditation): A Bigdata Approach",International Journal of Information Retrieval Research (IJIRR), 10(4),PP.1-
20,2020. doi:10.4018/IJIRR.2020100101
[12] G. Eason, B. Noble, and I. N. Sneddon, ``On certain integrals of Lipschitz-Hankel type involving products of Bessel
functions,'' Phil. Trans. Roy. Soc. London, vol. A247, pp. 529--551, April 1955.
[13] Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68--73.
[14] I S. Jacobs and C. P. Bean, ``Fine particles, thin films and exchange anisotropy,'' in Magnetism, vol. III, G. T. Rado and H.
Suhl, Eds. New York: Academic, 1963, pp. 271--350.
[15] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic
segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014, pp. 580–587
[16] Badino, H., Franke, U., Pfeiffer, D.: The stixel world-a compact medium level representation of the 3d-world. In: Joint Pattern
Recognition Symposium 2009. Springer, Berlin, Heidelberg
[17] Ahlstrom, T. Victor, C. Wege, and E. Steinmetz, “Processing of eye/head-tracking data in large-scale naturalistic driving data
sets,” IEEE Trans. Intell. Transp. Syst., vol. 13, no. 2, pp. 553–564, 2012.
[18] V. G. Dygalo and A. A. Revin, Technologies of testing of active safety systems of cars. Moscow, 2012.
[19] Kaminer I, Hallberg E, Silvestre C, et al.(1998) Trajectory tracking for autonomous vehicles: An integrated approach to
guidance and control[J]. Journal of Guidance, Control, and Dynamics: A Publication of the American Institute of Aeronautics
and Astronautics Devoted to the Technology of Dynamics and Control, 21(1):29-38.
[20] Qu T, Zhang Q, Sun S.(2016). Vehicle detection from high-resolution aerial images using spatial pyramid pooling-based deep
convolutional neural networks [J]. Multimedia Tools & Applications,1-13.
[21] P. Adarsh, P. Rathi and M. Kumar, “YOLO v3-Tiny: Object Detection and Recognition using one stage improved model,” in
6th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, pp. 687-
694, 2020, doi: 10.1109/ICACCS48705.2020.9074315.
[22] Dong Dong1, Xiao-Yang Li2, XiaoTong Sun3 School of Reliability and Systems Engineering, Beihang University Science
and Technology on Reliability and Environmental Engineering Laboratory Beijing, China- A Vision-based Method for
Improving the Safety of Self-driving.
[23] Haoran Song, College of engineering & computer science, Australian National University, "The Application of Computer
Vision in Responding to the Emergencies of Autonomous Driving." Canberra, Australia, 2600, Australia.
Siddhi Lahange has completed a Diploma in Computer Engineering from Dr. D.Y. Patil
polytechnic, Nerul. Currently pursuing Bachelor's Degree in Computer Engineering Final
year. Also working as an analyst intern at KPMG India, as an Analyst.
Deep Nayak is currently in his pre final year pursuing computer engineering at Sardar Patel
Institute of Technology. His interest lies in Machine Learning and Deep Learning
Algorithms and it's integration with Web Technologies
Atharva Mohite is currently in his pre-final year pursuing his Bachelor's degree in
Computer Engineering from Sardar Patel Institute of Technology, Andheri. Also working as
an intern in JP Morgan Chase in the Enterprise Architecture division
Mr. Pramod Bide is pursuing his PhD in Computer Engineering at the University of
Mumbai. He is an Assistant Professor at the Department of Computer Engineering, Sardar
Patel Institute of Technology, University of Mumbai. He has 10 years of experience in
Academic, Research and Administration. His research interests include social network
mining, Machine learning, data mining, data analytics.