5th IEEE International Conference on Signal Processing, Computing and Control (ISPCC 2k19), Oct 10-12, 2019, JUIT,

Solan, India

A Detection System for Stolen Vehicles Using Vehicle Attributes

With Deep Learning

Karambir Bidhan
Parveen Saini Sona Malhotra
Department of Computer Science &
Department of Computer Science & Department of Computer Science &
Engineering
Engineering Engineering
University Institute of Engineering and
University Institute of Engineering and University Institute of Engineering and
Technology
Technology Technology
(Kurukshetra University)
(Kurukshetra University) (Kurukshetra University)
Kurukshetra, India
Kurukshetra, India Kurukshetra, India
[email protected]
[email protected], [email protected]

Abstract— there is phenomenal growth in numbers of vehicles To deal with such situations this paper present a system to
on road as well as in numbers of theft cases. In a few countries, detect the stolen vehicle or suspicious vehicle without any
vehicle theft cases are believed to be one in every 10 minutes. In human-intervention which could also identify the vehicle's
such high-intensity scenario, it is not possible to identify the anomalies like modification in color or any tampering with
stolen vehicle with traditional methods such as manual check, number plate based on deep learning concepts. Particularly, a
RFID base technologies where it is easy to manipulate and stolen vehicle identification system that automatically
remove. This paper presents a design with the implementation of identifies the ongoing vehicle in real-time scenario according
a system based on deep learning technique where SSD algorithm to the identification of vehicle color and the identification of
is coupled with K-Nearest Neighbors algorithm and
vehicle registration number and further it can present important
convolutional neural network classifier, to identify the
stolen/suspicious vehicles without human intervention. This
information to the police for pointing suspect/stolen vehicles.
design operates in two stages: At first stage Tensor flow, Tensor Though there are many technologies consisting of innovative
Flow detection API from Google is used to detect the vehicle with methods to identify the moving object in the domain of
sub-modules to identify color and registration number from a computer vision, the Deep learning is to be proved extensively
real-time video stream. In second stage it compares the extracted effective both in industry and academics due to its notable
characteristic i.e. vehicle registration number and color with the performance[1].
RTO record. If in comparison it observes any anomaly in two sets
or matched with stolen vehicle complaint record at police station. Object detection is crucial and imperative tasks in the
It would notify the concerned authorities. A prototype model is territory of the computer’s ability to see. The major chore of
developed based on the presented design. object detection is to pinpoint the objects in the images,
distinguish the object class and reveal the position of each
object. However, due to various factors such as posture, angle
of image, occlusion, and perspective, images become distorted
Keywords—convolution neural network; computer vision; and it brings challenges in object detection.
TensorFlow; vehicle detection; Single Shot MultiBox Detector
(SSD) algorithm, KNN, Keras To detect the object effectively, in 2006 Hinton extracted
the better features with the conception of deep learning, by
I. INTRODUCTION using deep neural networks (DNN)[2]. There is implementation
it automatically learns high-level features from a gigantic
In the past couple of decades, the number of vehicles on amount of sample data. With constant development of deep
road has been increased with astronomical speed and the learning, numerous deep learning networks with diverse
number of stolen vehicles counts and their uses in crime functions have been proposed, such as Convolutional Neural
activities too. It has become crucial for security to capture these Network (CNN)[3] [4] [5], Multi-layer Perceptron (MLP),
vehicles. But in such a situation where roads are occupied with Recurrent Neural Network (RNN) and Generative Adversarial
high-density traffic, traditional methods like inspection of the Network (GAN). It has been observed in various researches
vehicle by halting the traffic become obsolete. Moreover, other that the anti-interference ability will be much better and the
automatic methods like radio frequency identification (RFID) accuracy can be greatly improved, by using convolutional
technology-based system, Barcode system, and GPS based neural network to detect the object. This is mainly due to the
system can be easily manipulated. In addition to it, it has been convolutional neural network extracts high-level features and
observed that criminals usually alter the vehicle color and the expressive ability of the features is improved. Moreover,
registration number to commit the crimes and this scenario convolutional neural networks perform feature extraction,
cannot be addressed by conventional approaches. feature selection, and feature classification together in the one

978-1-7281-3988-3/19/$31.00 ©2019 IEEE 251

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 01,2020 at 18:57:18 UTC from IEEE Xplore. Restrictions apply.
 5th IEEE International Conference on Signal Processing, Computing and Control (ISPCC 2k19), Oct 10-12, 2019, JUIT, Solan, India

model only, which will optimize the function of the network as III. ARCHITECTURE AND FLOW OF SYSTEM
a whole[6].
A. Flow Diagram of the system
In recent years, the algorithm models constructed with the
convolutional neural network have got good competition The basic flow of the system is that it read the source video
results in various deep learning challenges. Like the Yolo frame by frame till the end of the video and in the end generate
algorithm [7] and SSD algorithm [8]. In this paper SSD the vehicle output as stolen /suspicious vehicle or not.
algorithm is incorporated which uses the convolutional neural
network as the basic network for feature extraction in object
detection. Single Shot MultiBox Detector (SSD) is
comparatively faster and more robust among various object
detection methods because it uses multiple different
convolution layers for object detection.

II. FRAME WORK OF PROTOTYPE

A. Data Collection
First, Road traffic Real-time video is captured with the camera.
Fig.2. Flow Diagram of the System
B. Vehicle Detection, Color Recognisation and Vehicle
Registration Number Identification B. Vehicle Detection Module

This section has three sub-parts, firstly it detects the vehicle Object detection has always been one of the most imperative
from video and on that detected vehicle it recognizes the color domains of computer vision. Single Shot Multi-Box Detector is
and identifies the registration number of the vehicle. comparatively faster and more robust among various object
detection procedures.
C. Identification of stolen/suspicious vehicles In this system design, the Single Shot Multi-box Detector
Vehicle extracted data is compared with the Police / (SSD) model has been used to detect the moving vehicle. SSD
Regional Transport Office (RTO) record. In other words, the algorithms use the convolutional neural network (CNN) as the
extracted vehicle information form section B is compared with basic network. It basically uses different convolutional layers at
the police stolen vehicle database system. If here any anomaly each level. It extracts the features with variable sizes, and
is detected by the system like vehicle is listed in stolen subsequently calculates the confidence loss, location loss and
category record of police or vehicle color is different form gives the bounding box.
RTO record as in relation with vehicle registration number and
vice versa. Then it is reported as suspicious vehicle to the
police or concerned authorities. Subsequently, that particular
vehicle can be stopped by police alone without disturbing the
other traffic.

Fig.3. Architecture of Single Shot Multibox Detector

SSD algorithm multiple different convolution layers provide

the features at each level. So feature obtained from multiple
layers leads to fewer losses in features. Moreover, it does not
implement the full connection among the layers due to which
detection process becomes faster as compare to other models
and detection accuracy is much improved.

C. Vehicle Color Recognition Module

Fig.1. Block Diagram of the Design

978-1-7281-3988-3/19/$31.00 ©2019 IEEE 252

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 01,2020 at 18:57:18 UTC from IEEE Xplore. Restrictions apply.
 5th IEEE International Conference on Signal Processing, Computing and Control (ISPCC 2k19), Oct 10-12, 2019, JUIT, Solan, India

Detected vehicle then acts as input to the Vehicle Color phase, character is segmented and then recognized
Recognition Module. This module predicts the approximate Convolutional Neural Network classifier.
color of vehicles. To accomplish this task KNN[8] machine
learning classification algorithm used which is trained on R, Further to predict the approximate color of vehicles, KNN
G, B color histogram features. Machine Learning Classification Algorithm is used which is
trained on color histogram features. This setup can identify the
D. Vehicle Registration number Recognition Module
violet, green, white, yellow, red, blue, black and, orange
colors. To recognize the vehicle color it performs the task in
The vehicle registration number recognition module works in two steps, at first it extracts the features and then classifies the
three major phases- number plate localization, character color with K-Nearest Neighbors algorithm. For feature
segmentation, and recognition. This prototype task of extraction, color histogram calculation is performed with
registration number identification is performed by the OpenCV and NumPy is used for performing the computation
Convolutional Neural Network classifier. of Euclidean distance and to get the k nearest neighbors.

. V. RESULTS AND ANALYSIS

IV. EXPERIMENTAL SETUP AND DATASETS For validating the system we have considered vehicle dataset as
shown in figures (The sample frames used for testing). The
Basically, every video frame here is feed to the “SSD with figure shows the detected vehicle with color information and
Mobilenet” model, which is based on CNN as a basic network registration number.
and developed with Google’s Tensorflow.
We have two sub-modules here. One is to detect the vehicle's
features like vehicle color and another is to identify the
vehicle number from the number plate. For both modules
input (which is detected and classified vehicles) is received
from same source i.e. output of the TensorFlow Object
Detection API.
To begin with, source video is read by OpenCv and it
processes the complete video frame by frame till end. At the
next level TensorFlow, Object detection API uses the frame to
detect and classify the vehicles.
Here we have used the “SSD with Mobilenet” model form the
TensorFlow model zoo which has been pre-trained with the Fig.4. Vehicle color white is detected
COCO dataset [10]. COCO dataset has 80 object categories,
200k labeled images and every image has five captions. More
precisely, we have used ssd_mobilenet_v1_coco model from
the pool.

In the current setup to identify the vehicle registration number, Fig.5. Recognized Vehicle registration number
it uses the pre-trained Convolutional Neural Network
classifier. This classifier is implemented using the Keras and
training is accomplished with 47605 images of 36 classes
which are ranged from 0-9 and A-Z. Furthermore, the ReLu
activation function is used at convolution layers and softmax
function is employed at last prediction layer. It accepts input
images of size 64x64 at the first convolutional layer and uses
the Root Mean square optimizer. As it has three phases. In
first phase, it processes the received frame or input image into
a grayscale image because grayscale images have less
computation cost. In second phase, automobile number plate is
localized from the automobile image with the openCv
findcontour function. This function detects all boundaries and Fig.6. Vehicle color blue is detected
pulls out the character image from grayscale image. In last

978-1-7281-3988-3/19/$31.00 ©2019 IEEE 253

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 01,2020 at 18:57:18 UTC from IEEE Xplore. Restrictions apply.
 5th IEEE International Conference on Signal Processing, Computing and Control (ISPCC 2k19), Oct 10-12, 2019, JUIT, Solan, India

The current setup is able to identify only discrete colors. So

there is lots of work needs to be done to identify vehicles with
multiple colors. Further, it has shown the false-positive result
for images which are captured in bad lighting or shadow.
Therefore, need to put some filtering method before KNN
algorithm.

REFERENCES
[1] O. Russakovsky et al., “ImageNet Large Scale Visual
Fig.6. Vehicle color yellow is detected Recognition Challenge,” Int J Comput Vis, vol. 115, no.
3, pp. 211–252, Dec. 2015.

[2] G. E. Hinton, “Reducing the Dimensionality of Data with

Neural Networks,” Science, vol. 313, no. 5786, pp. 504–
507, Jul. 2006.

[3] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A.

Torralba, “Object Detectors Emerge in Deep Scene
CNNs,” arXiv:1412.6856 [cs], Dec. 2014.

[4] D. G. Lowe, “Distinctive Image Features from Scale-

Invariant Keypoints,” International Journal of Computer
Vision, vol. 60, no. 2, pp. 91–110, Nov. 2004.

Fig.7. Vehicle color red is detected [5] A. Krizhevsky, I. Sutskever, and G. E. Hinton,
“ImageNet classification with deep convolutional neural
networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90,
The results of the designed system are satisfying. It is able to May 2017.
detect the vehicle color and registration number successfully.
[6] J. R. R. Uijlings, K. E. A. Van De Sande, T. Gevers, and
VI. CONCLUSION
A. W. M. Smeulders, “Selective search for object
This paper has provided a more robust utility to identify the recognition,” International Journal of Computer Vision,
stolen/suspicious vehicles without human intervention by vol. 104, no. 2, pp. 154–171, 2013.
validating through multiple vehicle attributes instead of just
one i.e. registration number. [7] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You
At the same time, it has demonstrated the convenience and Only Look Once: Unified, Real-Time Object Detection,”
feasibility to develop a tailor-made system that uses a in 2016 IEEE Conference on Computer Vision and
combination of deep learning methodologies to provide an Pattern Recognition (CVPR), Las Vegas, NV, USA,
automated solution to human complex tasks. 2016, pp. 779–788.

[8] W. Liu et al., “SSD: Single Shot MultiBox Detector,” in

VII. FUTURE WORK
Computer Vision – ECCV 2016, vol. 9905, B. Leibe, J.
There is a scope to further improve the system by adding more Matas, N. Sebe, and M. Welling, Eds. Cham: Springer
features like detection of suspicious vehicle by identifying the International Publishing, 2016, pp. 21–37.
driver face expressions (anxiety) and by identifying over-
speed of vehicles. [9] S. Zhang, S. Member, X. Li, M. Zong, X. Zhu, and R.
Wang, “Efficient kNN Classification With Different
The presented model is based on a fully-supervised training Numbers of Nearest Neighbors,” pp. 1–12, 2017.
method that has a limitation with amount of data. As data
grows, it is not possible to label this ever-growing dataset. So [10] T. Y. Lin et al., “Microsoft COCO: Common objects in
further there is need to find how we can further exploit the context,” Lecture Notes in Computer Science (including
power of CNNs with unsupervised detection or weakly subseries Lecture Notes in Artificial Intelligence and
supervised methods. Lecture Notes in Bioinformatics), vol. 8693 LNCS, no.
PART 5, pp. 740–755, 2014.

978-1-7281-3988-3/19/$31.00 ©2019 IEEE 254

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 01,2020 at 18:57:18 UTC from IEEE Xplore. Restrictions apply.