Number Plate Detection & Recognition System

Department of Computer Science & Engineering

Bachelor of Technology(Computer Science & Engineering)
Semester – V , Section - C

Project Guide :
Nazmul Arefin
DCS Mentor

SUBMITTED BY :

Team :- 5CT3

Team : 5CT3
1 Introduction 1
 1.1 Automatic Number Plate Detection 1
1.2 Motivation 1
1.3 Problem Statement 2
1.4 Innovative Idea for our Project 2
2 Literature Survey 3
2.1 Automatic Number Plate Recognition Literature Survey 3
2.2 Research Gap 5
3 Requirement Gathering 6
3.1 Number Plate Detection Data 6
3.2 Character Recognition Data 6
4 Module Description 7
4.1 Number Plate Detection 7
4.1.1 YOLO 7
4.1.2 Contour Method 8
4.2 Character Segmentation 8
4.3 Character Recognition 9
5 Coding and Testing 10
5.1 Code Snippets for Number Plate detection using YOLO 10
5.2 Code Snippets for Number Plate detection using 11
Contour
5.3 Code Snippets for Character Segmentation 12
5.4 Code Snippets for Character Recognition 12

5.5 Result 13
6 Experimental Analysis 14
7 Conclusion and Future Work 15
Reference
 ABSTRACT

India is one of the world's most densely inhabited countries having a lot of
vehicles. Therefore, a proper traffic management system to correctly detect
automobiles is necessary.
We will extract the vehicle registration number from the license plate of
different Indian Automobiles using Deep learning techniques. We will develop a
real time model that inputs automobile images and displays the registration
number of the vehicle. Initially, we start with detecting the vehicle image from
which the region of interest is extracted i.e. the License plate using hybrid
method i.e. combining YOLO and contour method. In order to accomplish
character segmentation, a bounding box is made around each character. Next,
the registration number is extracted for character recognition using CNN. The
registration number is then displayed as an output.
We will be creating a custom data set consisting of images of different automobiles
in different conditions and orientations to train our model.
ANPR can be used to correctly monitor vehicles at college entrances and other
locations with rigorous access controls, by police forces to identify criminal
activity, and on highways as a method of identifying speeding through average
speed calculation.
 CHAPTER 1 INTRODUCTION
Automatic Number Plate Recognition system detects the Region of Interest i.e. the
license number plate and then extracts the vehicle registration number from it. It is
helpful for upholding the law, determining whether a vehicle is registered, identifying
speeders, managing parking spaces, preventing vehicle theft, and assisting authorities
in identifying and tracking down moving automobiles.

1.1 Automatic Number Plate Recognition

The highly accurate Automatic Number Plate Recognition (ANPR) technology can
read vehicle license plates without the need for human involvement.
YOLO, also known as You Only Look Once, is one of the most powerful real-time
object detection algorithms. It detects and classifies the whole image at once using
a single convolutional network that predicts the bounding box and class
probabilities. For a particular object multiple bounding boxes are created with
corresponding class probabilities IOU (Intersection over Union) for all the
bounding boxes generated for a particular class are compared and the one with the
maximum IOU is selected.
Contour method follows a set of procedures - resizing the image to 500px width,
converting to grayscale, removing noise, binarizing, finding contours and drawing
contour.
Convolutional neural networks are a popular deep neural network architecture
which can be used for implementing Optical Character Recognition. It learns from
the important 2D features present in the image for classification

1.2 Motivation

The motivation behind the project is to:

1
 Uphold law and order: In India, excessive speeding is the primary cause of
fatalities. ANPR can be used to track the average speed of the cars and spot any that
go over the speed limit.

Parking management - Vehicles with registered license plates can enter parking
lots without paying, while unregistered vehicles must pay at the time of check-in
and check-out. Parking tickets can be paid directly from the user's account against
the produced ticket number using the number plates of the automobile that are
directly linked to the owner's mobile phone.

Preventing vehicle theft - ANPR may be used to track vehicles, which can help to
prevent auto theft. Law enforcement will be able to determine the exact moment,
place, and path taken by a stolen car.

1.3 Problem Statement

To develop a real-time model that captures the image of a vehicle provided to

detect the number plater and extract the registration number from it.

1.4 Innovative idea for our project

The project is divided into two phases - Number Plate Detection and Character
Recognition. We will be using the hybrid approach i.e. YOLO algorithm and
contour method for Number plate detection and training a CNN model for
Character recognition.

We will be creating our own custom dataset consisting of images of different

automobiles in different conditions and orientations.

2
The remaining sections of this paper are systematized as follows – chapter 2 provides a
detailed literature survey of existing methodologies for ANPR and their findings,
chapter 3 mentions the research gaps found in the papers studied and chapter 4 details
the

CHAPTER 2: LITERATURE SURVEY

In this digital era, research in automatic number plate detection is very important for
its uses in several areas. For example, it can be used at places with strong access
restrictions to monitor vehicles properly, by police forces to track down criminal
behavior and to detect speeding through average speed calculation.

2.1 Automatic Number Plate Recognition Literature Survey

[1] used Image Processing Techniques like Color Image Processing, Contrast

Extension, Adaptive Thresholding, Median Filtering, Segmenting Characters for

plate detection & Template Matching using OCR for text extraction. For plates at
a specific angle and plates near the edge of the photograph, it did not yield the
required results. Character segmentation was successful in circumstances where
82.6% of the letters could be identified.

[2] used image processing techniques for plate detection and an algorithm for

template matching to segment and identify the characters. If English characters

cannot be accord with characters, Hindi characters from the established template
are used instead. They found that taking a photograph from the side and with low
brightness and contrast has an effect on the results. There were 97% and 98%
average segmentation rates (SR) and recognition rates (RR), respectively.

3
[3] used Image Processing Techniques for plate detection and Template matching

using OCR algorithm for text extraction. Number Plate Extraction success rate
was found to be around 85% and Character Segmentation success rate 80%.

[4] used Faster-RCNN with ResNet-50 model and VGG16 separately for training

the network. Character Segmentation (CS) was done using ResNet-50 as Base-
CNN. Standard CNN model was used for Character Recognition. They discovered
that the precision of license plate detection was 94.98%. At an average mean
accuracy of 99.55 %, more than 99% of characters were successfully segmented,
and the pipeline correctly identified 98.6% of the segmented characters. [5]
trained a Faster-RCNN model using TF Object Detection API. Contrast was
maximized using Top and Black hat morphological transformations. Character
segmentation and recognition was done using Tesseract OCR. The proposed work
was sensitive to changes of light intensity in the surroundings of the number
plates. The Respro optimizing algorithm performed best at a learning rate of 0.
002.

[6] compared two methodologies - YOLO v3 and Traditional Image Processing.

The accuracy, recall, precision for traditional image processing technique was
found to be 72%, 0.92, 0.92 whereas for YOLOv3 it was 90%, 0.92 and 0.92
respectively.

[7] carried out Plate Segmentation using image processing techniques, Plate

Classification using SVM and Plate Recognition using OCR Segmentation

and OCR Classification i.e. MLP (Multi Layered Perceptron). The suggested
method successfully identified several vehicle licenses plates in a single
video frame. Even license plates with intricate backgrounds can be
successfully traced.

4
 [8] used Image Processing Techniques for Plate detection and compared

classification of characters from license plate using 1NN and ANN. Plate
Recognition accuracy of 1NN was found to be 87.43% and that of ANN was
86.34%. As the "k" value is raised, fewer characters are accurately identified.
For wrongly identified characters, the opposite is accurate.

[9] used Image Processing Techniques for Plate Extraction, Bounding Box for

Character Segmentation and template matching for Character Recognition.

Character Segmentation, Plate Extraction, and Character Recognition had an
accuracy of 86.67% , 93.33%, and 93.33% respectively.

[10] used custom YOLO for Plate Detection and Character Recognition. The
Number Plate
Detection accuracy was found to be 100% and Number Plate Recognition was
91%

[11] used Wiener filtering, YOLOv3 for License Plate detection and OCR for

image enhancement and recognition. Accuracy for number plate detection

using YOLO model was found to be 96% and the accuracy for character
recognition using OCR model was found to be 91.5%

[12] employed YOLO v3 for Number Plate Recognition and CRNN for Number

Plate Classification. The suggested technique recognized 88% of 3-letter

plates and 99 % of 4-letter plates, yielding an 86% recognition rate.
After doing a detailed literature survey on Automatic Number Plate Recognition we
found a few limitations in the above-mentioned Research papers. The detection was
dependent on a few factors such as weather condition, orientation of the image,
distance of the number plate in the image and font.

5
2.2 Research Gap
In most of the literature survey done, it can be identified that the detection is
dependent on a few factors such as weather condition, orientation of the image,
distance of the number plate in the image and font making recognition difficult and
less accurate.

1. Weather conditions – According to the weather conditions the image might

contain raindrops, fog, etc.

2. Rotated images - Images of license plates taken from some angle.

3. Farther images - Images taken from a distance having a smaller area of

license plate.

4. Misplaced Characters - Characters having similarities or looking alike

such as 8 and B, 5 and S, etc.

We will improve upon these limitations by creating a custom dataset to train the
model in all aspects.

REQUIREMENT GATHERING :

3.1 Number Plate Data

We have gathered a dataset of 4954 images by combining datasets from different
sources - dataset containing 603 images of vehicles of different Indian states
annotated in PASCAL V2 format then converted to YOLO format as shown in
figure 3.1 from Kaggle by Sai Sirisha N, Pranav Kant Gaur and Vikshipt, dataset
containing 4191 images of vehicles of Brazil annotated in YOLO format and
dataset containing 160 images captured from a video taken in the state of Odisha
from Kaggle by Deepak

6
 Figure 3. 1 Dataset for YOLO

3.2 Character Recognition Data

We gathered a dataset having a total of 862 images as shown in figure 3.2 for
training and 216 images for validation of 36 classes (0-9 and A-Z) from Kaggle
for character recognition.

Figure 3. 2 Dataset for CNN

MODULE DESCRIPTION :

The Automatic Number Plate Detection System consists of 5 modules as shown in

Figure 4.1. They are as follows:
1. Image Acquisition: Gathering the dataset required for training the YOLOv3

model and CNN model.

7
 2. Image Pre-processing: Annotating the images in YOLO format for YOLO

model training and creating classes for CNN model training.

3. Number Plate detection: Detecting number plate from vehicle images using the

trained YOLO model and contour method (Image Processing techniques).

4. Character Segmentation: Segmenting characters from the detected number

plate using image processing techniques

5. Character Recognition: Recognizing characters obtained from segmentation

using trained CNN model.

Figure 4. 1 System Architecture

4.1 Number Plate Detection

4.1.1 YOLO
An image is divided into feature vectors having the following properties -
Pc(presence of object), Bx (abscissa of center of B.B), By (ordinate of center
of B.B), Bw (width of the B.B), Bh (height of the B.B) and C1, C2 (class
probabilities).
The image is divided into m x m grids with feature vectors of their own
displaying the corresponding values only if the centroid of the object is
present in the particular grid creating the final feature vector of m x m x w (w
denoting size of feature vector of one grid) dimension.

For a particular object multiple bounding boxes are created with

corresponding class probabilities IOU (Intersection over Union) for all the
bounding boxes generated for a particular class are compared and the one with

8
 the maximum IOU is selected. The prediction of Number Plate by YOLO
model is shown below in Figure 4.2

Figure 4. 2 License Plate detection using YOLO

4.1.2 Contour Method

Number Plate detection by contour method follows a set of procedures as
depicted in Figure 4.3. These steps include - resizing the image to 500px
width, converting to grayscale, removing noise, binarizing, finding contours
and drawing contour if it is in quadrilateral form with area greater than 30.

Figure 4. 3 License Plate Detection using Contour Method

4.2 Character Segmentation

The following image processing techniques are used for segmentation of characters
from the detected license plate – Gray Scale Conversion, Thresholding for
converting to binary image, eroding for noise removal, Dilation to fill up absent

9
 pixels, creating a list of dimensions and comparing them to create the smallest box
possible around each character. The segmentation of individual characters by
contour method is depicted below in Figure 4.4

2
Figure 4. 4 Character Segmentation using Image Processing Techniques

4.3 Character Recognition

Character Recognition is carried out using a trained CNN model. The model as
shown in Figure 4.5 by extracting features from a character, making it into a filter
and sliding it over the entire image to create a feature map.
Two main parts of the CNN architecture -
• A convolution tool with several pairs of Convolution+ReLu and pooling layers
that isolates and identifies the distinct features of the image for feature
extraction.
• A fully connected layer that utilizes the results of the convolutional process and
establishes the image class utilizing the features that were previously extracted.

10
 Figure 4. 5 Predicted Characters using CNN

CODE AND TESTING :

5.1 Code Snippets for Number Plate detection using YOLO

Figure 5. 1 Parameters for YOLO prediction

11
 Figure 5. 2 Functions used in YOLO

Figure 5. 3 Number Plate detection using YOLO

5.2 Code Snippets for Number Plate detection using Contour Method

12
 Figure 5. 4 Image Processing Techniques used in Contour Method

5.3 Code Snippets for Character Segmentation

Figure 5. 5 Character Segmentation using Image Processing Techniques

13
5.4 Code Snippets for Character Recognition

Figure 5. 6 Character Recognition using CNN Trained Model

5.5 Result

Figure 5. 7 Prediction using YOLO + CNN

14
 Figure 5. 8 Prediction using Contour method + CNN

Figure 5. 9 Prediction using Hybrid Method

EXPERIMENTAL ANALYSIS :

The YOLO and CNN models were trained using Alexey AB darknet and TensorFlow
Framework respectively in NVIDIA-SMI GPU with CUDA integration.
The trained YOLO model gave an average loss of 0.166, average IOU of 0.886 in
1600 epochs.
The CNN model trained for 25 epochs gave results as in Table 1.

Accuracy Loss

Training 0.9826 0.0480

Validation 0.9630 0.0785

15
 Table 6. 1 Training Curve of CNN model

The CNN training curve with accuracy and loss of training set and validation set is depicted in
Figure 6.

Figure 6. 1 CNN training and validation set curve

CONCLUSION AND FUTURE WORK :

The number plate detection using YOLO with CNN on our test dataset gave an
accuracy of 70% and that of Contour Method with CNN gave an accuracy of 63.6%.
The Hybrid model combining these two methods i.e. YOLO and Contour for Plate
recognition n and CNN for character recognition gave us an accuracy of 90.91%.
Images which are closer and at an angle are better detected by Contour method.
The YOLO approach was used to alleviate the problem of the contour method having
trouble detecting images that are farther away or have number plate colors that are
similar to that of the vehicle. Consequently, utilizing a hybrid technique produced
greater outcomes than using each one separately.

16
Provided that the project is further worked upon, we can implement real-time number
plate recognition that has a lot of use-cases. For example, it can be used as an added
security feature. The awareness that a person's license plate is being recorded and examined is
sufficient to deter criminal activity in advance.
It can be improved to become an automated service. If the project can achieve high accuracy
readings, it can be used to monitor parking solutions in an effective and affordable manner. In the
past, it took time to record number plates, and it took much longer to notify offenders of traffic
regulations of their penalties.
However, with the help of this effort, number plates can be quickly identified and compared to the
database. Like FASTAG, we can implement number plate recognition while cars are in motion that
can be beneficial in numerous sectors of the industry.

17