See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/327099013

CNN-RNN based method for license plate recognition

Article · July 2018

DOI: 10.1049/trit.2018.1015

CITATIONS READS

9 1,398

6 authors, including:

Palaiahnakote Shivakumara Dongqi Tang

National University of Singapore Nanjing University
138 PUBLICATIONS   1,742 CITATIONS    2 PUBLICATIONS   9 CITATIONS   

SEE PROFILE SEE PROFILE

Tong Lu Umapada Pal

Nanjing University Indian Statistical Institute
148 PUBLICATIONS   695 CITATIONS    396 PUBLICATIONS   6,275 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Document image retrieval View project

CNN on Degraded Images View project

All content following this page was uploaded by Hossein Anisi on 31 August 2018.

The user has requested enhancement of the downloaded file.

CAAI Transactions on Intelligence Technology

Research Article

ISSN 2468-2322
CNN-RNN based method for license Received on 20th July 2018
Accepted on 21st July 2018
plate recognition doi: 10.1049/trit.2018.1015
www.ietdl.org

Palaiahnakote Shivakumara 1 ✉, Dongqi Tang 2, Maryam Asadzadehkaljahi 1, Tong Lu 2,

Umapada Pal 3, Mohammad Hossein Anisi 4
1
Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia
2
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, People’s Republic of China
3
Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, Kolkata, India
4
School of Computer Science and Electronic Engineering, University of Essex, CO4 3SQ Colchester, UK
✉ E-mail: [email protected]

Abstract: Achieving good recognition results for License plates is challenging due to multiple adverse factors.
For instance, in Malaysia, where private vehicle (e.g., cars) have numbers with dark background, while public vehicle
(taxis/cabs) have numbers with white background. To reduce the complexity of the problem, we propose to classify the
above two types of images such that one can choose an appropriate method to achieve better results. Therefore, in
this work, we explore the combination of Convolutional Neural Networks (CNN) and Recurrent Neural Networks
namely, BLSTM (Bi-Directional Long Short Term Memory), for recognition. The CNN has been used for feature
extraction as it has high discriminative ability, at the same time, BLSTM has the ability to extract context information
based on the past information. For classification, we propose Dense Cluster based Voting (DCV), which separates
foreground and background for successful classification of private and public. Experimental results on live data given
by MIMOS, which is funded by Malaysian Government and the standard dataset UCSD show that the proposed
classification outperforms the existing methods. In addition, the recognition results show that the recognition
performance improves significantly after classification compared to before classification.

1 Introduction neural networks (CNNs) and recurrent neural networks to address

the above mentioned-challenges.
Despite sophisticated license plate recognition systems available
online [1], the problem of license plate recognition is considered
as a hot research issue. This is due to the lack of uniform format 2 Related work
of capturing and displaying number plates across countries. For
example, when we look at developed countries, all vehicles follow Since the proposed work involves classification of different types
a particular format or type for displaying number plates. As a of license plate images and recognition of license plate images,
result, the developed systems work well for images of respective we review the methods on classification and recognition here. As
countries. However, for the countries like Malaysia and India, one per our knowledge, there are no methods for classification of
can see many types of number plates with different texts. One different license type images. However, one can see several
such example is shown in Fig. 1, where it can be seen in Fig. 1a methods for classification of caption and scene text in video. It is
that private and UCSD have dark background and white foreground, true that number plates can be considered as scene text images as
while public has opposite. It is also observed in Fig. 1a that images they are captured from natural scenes. Therefore, we present the
of public suffer from poor quality due to degradations and dis- review of caption and scene text classification methods in the
tortions. For these images, when we apply the existing method [2] following. Raghunandan et al. [4] proposed new sharpness
which proposes strokelets for character identification and then features for image type classification based on textual information.
recognition of scene texts using a random forest classifier, it does The method works based on the fact that caption text images have
not recognise public and UCSD correctly as shown in Fig. 1b. high quality and clarity because is they are edited texts, while the
This is due to the change in background and foreground colour nature of scene texts is unpredictable because it is a part of an
with the distortion effect. The online system, which explores deep image. Xu et al. [5] proposed caption and scene text classification
learning for Chinese license plate recognition [3] works well for in video, which uses gradient and clustering. The method works
private and public but fails for UCSD images as shown in Fig. 1c. on the basis of pixels in caption text images share the uniform
The reason for the poor results is that UCSD suffers from poor colour value, while scene text does not. Shivakumara et al. [6]
quality due to blur. It is noted from Fig. 1 that the method [2] proposed the separation of graphics and scene texts in video
which works well for natural scene texts and the system [3] which frames, which explores the contrast and characteristics of text
uses deep learning fail to perform well. components. Since caption texts have good quality, it is expected
Therefore, we can conclude that the existing powerful methods that they should exhibit regular patterns of spacing between their
and systems may work well for images of particular types but not characters, while scene texts do not. Roy et al. [7] proposed new
images affected by multiple adverse factors as shown in Fig. 1a, tampered discrete cosine transform (DCT) features for scene and
and hence the latter is challenging. Thus, in this work, we propose caption text classification in video frames for classifying two types
to classify images of different types such that we can focus on of texts in video. The method explores these tampered features
addressing other challenges, like effect distortion, noise and poor because editing texts over video is a type of tampering. It is noted
quality. In this work, we explore the combination of convolutional from the above review that the methods do not focus on

CAAI Trans. Intell. Technol., pp. 1–7

This is an open access article published by the IET, Chinese Association for Artificial Intelligence and 1
Chongqing University of Technology under the Creative Commons Attribution-NoDerivs License
(http://creativecommons.org/licenses/by-nd/3.0/)
 3 Proposed methodology
In this work, we consider segmented license plates as the input
for recognition. There are methods that detect license plates irres-
pective of the above-mentioned challenges because segmentation is
a pre-processing step for recognition [1]. The proposed method
consists of two parts, namely, part 1 presents the classification of
private and public license plate images, and part 2 presents the
recognition of license plate images. As discussed in Section 1, in
Malaysia, license plates of privates have a dark background, while
publics have a white background. It is true that values of white
pixels in images have high values compared to the values of dark
pixels. Based on this, we propose to separate foreground and back-
ground information through a Canny edge image of the input image.
Fig. 1 Illustrating the complexity of the license plate recognition Edge pixels in the Canny edge image are considered as foreground in-
a Input license plate images with different complexities
formation, and the rest pixels are considered as background pixels. The
b Recognition result of the existing method [2] proposed method applies K-means clustering with K = 2 on foreground
c Recognition result of the existing system [3] and background separately, which gives two clusters for foreground
d Recognition result of the proposed method and two more clusters for the background. The statistical analysis of
the clusters of foreground and background results in two hypothesises
for classification of private and public license plate images.
For the classified license plate images, we propose the com-
addressing the issues of adverse multiple factors such as blur, bination of CNN and RNN, which involves BLSTM (bi-directional
noise and colour change in background and foreground. Therefore, long-short term memory), for recognition. The proposed method
these methods may not perform well for license plate image extracts features using CNN as it has the ability to provide unique
classification. features for each character and use BLSTM for finding the sequence
In the same way, we also present a review on license plate recog- of labelling as it has the ability to define context using the past and
nition using deep learning in the following. Yang et al. [8] proposed the present features. Further, the proposed method recognises license
Chinese vehicle license plate recognition using a kernel-based plates based on the highest probability scoring with trained samples.
extreme learning machine with deep convolutional features. The More details can be found in the subsequent sections.
method uses CNN for feature extraction and then kernel based
extreme learning machines (KELM) for recognition. The main
objective of this method is Chinese license plate recognition. 3.1 Classification of license plates
Abedin et al. [9] proposed a license plate recognition system based
on contour properties and a deep learning model. The method seg- It is observed from Fig. 1a that the background and foreground of
ments characters for recognition. However, segmenting characters private plates are represented by black and white colours, respec-
accurately may be good for images which have high quality but tively. Similarly, background and foreground of public plates are
not images which suffer from poor quality. Polishetty et al. [10] pro- represented by white and black colours, respectively. This shows
posed a next generation secure cloud based deep learning license that the intensity values of the background of a normal plate are
plate recognition method for smart cities. The method involves lower than those of a taxi plate. In addition, it is also true that the
edge detection and binarisation for achieving results. It is noted number of pixels which represent background usually is higher
that edge detection and binarisation methods are sensitive than that of pixels which represent the foreground. These observa-
to complex background and distortions. Montazzolli and Jung [11] tions lead to propose a dense cluster-based voting (DCV) for the
proposed rea-time Brazilian license plate detection and recognition classification of private and public license plate images.
using deep CNNs. The method uses CNN for character segmentation It is true that the Canny edge detector gives fine edges regardless
and recognition. The method is proposed for high resolution of background and foreground colour changes as shown in Fig. 2a,
and quality images. Bulan et al. [12] proposed segmentation and where we can see edges are represented by white pixels for
annotation free license plate recognition with deep localisation both images. To extract the above observation, we separate edge
and failure identification. The method explores hidden Markov pixels as foreground and non-edge pixels as background for the
model (HMM) for character segmentation and recognition, which input image. Then the proposed method extracts intensity values
does not require specific character segmentation. However, it is not corresponding to foreground and background pixels from the grey
clear that how the method takes care of distortion effect, poor image of the input image. To visualise the difference in intensity
quality images and noises. Selmi et al. [13] proposed a deep learning distribution, we perform a histogram operation on intensity values
system for automatic license plate detection and recognition. The of foreground and background of the normal and taxi plate images
method performs many pre-processing steps before applying as shown in Figs. 2b–e. It is observed from Figs. 2b and c that the
CNNs. In addition, the method follows the conventional way of char- dense distribution can be seen for the pixels which have intensity
acter segmentation and then recognition. This idea is good for values near to 255 in the case of the foreground-normal plate,
images that have high quality and particular types. Li et al. [14] pro- while the dense distribution can be seen for the pixels which have
posed reading car license plates using deep neural networks. The intensity values near to 0 in case of background-normal plate
method explores the combination of CNN and RNN for license image. It is vice versa for foreground-taxi and background-taxi as
plate detection and recognition. However, it is not clear that how shown in Figs. 2d–e. This is the main basis for the proposing DCV.
the method works well for images affected by multiple adverse To extract such observations, we divide the intensity values of fore-
factors. ground and background into two clusters as Max which gets high
In summary, the review of license plate recognition using deep values and Min which gets low values using K-means clustering
learning shows that most of the methods focus on particular types with K = 2. This results in four clusters for the input image, namely,
of images with the knowledge of country or format of license foreground-Max cluster, Foreground-Min cluster, Background-Max
plates. None of the methods addresses the issues of blur, noise, fore- cluster and Background-Min cluster. For each cluster, the proposed
ground and background colour changes, and poor quality. Hence, in method computes the mean, standard deviation and the number of
this work, we tackle the above-challenges through classification pixels (density) to derive hypotheses to identify license plate
of different types of license plate images and then explore the com- images. For example, the product of standard deviation and the
bination of CNN and RNN in a different way for addressing other number of pixels of the background-min cluster is greater than that
causes to achieve the results. of standard deviation and the number of pixels of the background-max

CAAI Trans. Intell. Technol., pp. 1–7

2 This is an open access article published by the IET, Chinese Association for Artificial Intelligence and
Chongqing University of Technology under the Creative Commons Attribution-NoDerivs License
(http://creativecommons.org/licenses/by-nd/3.0/)
 Fig. 3 Foreground and background separation using edge pixels
a Foreground of the normal and taxi plate images
b Background of the normal and taxi plate images
c Grey values for the foreground pixels of the normal and taxi plate images
d Grey values for the background pixels of the normal and taxi plate images

Fig. 4 Min and Max clusters for foreground and background images of
Fig. 2 Intensity distribution of foreground and background of normal and normal and taxi license plate images
taxi plate images a Min and Max clusters for the foreground of normal plate image
b Min and Max clusters for the background of the normal image
a Canny images of normal and taxi plate images
c Min and Max clusters for the foreground of taxi image
b Histogram for grey of the foreground of the normal plate
d Min and Max clusters for the background of taxi image
c Histogram for grey of the background of the normal plate
d Histogram for grey of the foreground of taxi plate
e Histogram for grey of the background of taxi plate


Gx,y , if Canny x, y = 0
cluster for normal plates. This results in response ‘1’. In this way, the GBx,y = (2)
proposed method derives three hypotheses and finds the responses. If 0, else
the hypothesis gives two responses as ‘1’ out of three, it is identified as
a normal plate else a taxi plate. The proposed method applies K-means clustering with K = 2 on
The foreground and background separation is illustrated in Fig. 3, intensity values of foreground and background of private and
where Figs. 3a and b of private and public images provide edges public plate images to classify pixels which have high intensity
without losing shapes. Then the proposed method extracts intensity values into Max cluster and pixels which have low intensity values
values in the grey image G corresponding to foreground and back- into Min cluster as shown in Figs. 4a–d, respectively. It is noted
ground pixels, say GF and GB as defined in (1) and (2) for private from Figs. 4a and b that the number of pixels classified into the
and public plate images as, respectively, shown in Figs. 3c and d, Min cluster is higher than that of the Max cluster. Though the
where we notice there are colour changes in foreground and back- Max cluster gets high values, the number of pixels in the cluster is
ground of private and public plate images lower than that of pixels in the Min cluster. Therefore, the number

 of pixels in the cluster is considered as the weight, which is
Gx,y , if Canny x, y = 1 multiplied by the standard deviation as defined in (3). On the other
GFx,y = (1) hand, it is noted from Figs. 4c and d that the number of pixels
0, else which are classified into the Min cluster is lower than that of the

CAAI Trans. Intell. Technol., pp. 1–7

This is an open access article published by the IET, Chinese Association for Artificial Intelligence and 3
Chongqing University of Technology under the Creative Commons Attribution-NoDerivs License
(http://creativecommons.org/licenses/by-nd/3.0/)
Max cluster. This cue helps us to derive a hypothesis using the
number of pixels in clusters and the standard deviations to classify
private and public plate images.
The hypotheses H-1, H-2 and H-3 are derived as defined in
(4)–(6), where FNmin and FNmax denote the number of pixels in
min and max clusters of the foreground, respectively, while BNmin
and BNmax denote the number of pixels in background-min and
max clusters, respectively. BStdmin and BStdmax denote standard
deviations of the background of min-cluster and max cluster, respec-
tively. The proposed method considers each response of the hypo-
thesis as ‘1’ if it satisfies the condition, else it is considered as a
response ‘0’. Out of the three responses, if two responses are ‘1’,
the input image is identified as a normal one; else it is a taxi
image. Therefore, if two responses are ‘0’, the image is identified
as a taxi. In this way, the proposed method tests all the eight com-
binations of three responses for the input image. This process is
called as voting as defined in (7), where ∂ is the majority variable,
which is set to be greater than or equal to 2 for private and less
than 2 for the public


  2

m
 g=1 Mj − Xg
Stdj = (3)
m

where Mj is the mean of the j cluster, X denotes intensity values, and

m is the total number of the pixels in cluster j.

1 if FNmin . FNmax
H-1 = (4)
0 else

1 if BNmin . BNmax
H-2 = (5)
0 else Fig. 5 Proposed architecture of license plate recognition

1 if BNmin ∗BStdmin . BNmax ∗BStdmax
H-3 = (6)
0 else

1 if (H-1) + (H-2) + (H-3) . ∂
Voting = (7)
0 else

3.2 LSTM for license plate recognition

The proposed architecture of license plate recognition is shown Fig. 6 CNN is used for feature extraction
in Fig. 5, where we can see there are three steps, namely, CNN for
feature extraction, LSTM for extracting sequence, and connectionist
temporal classification (CTC) for the final recognition.
The architecture for CNN is derived from visual geometry group
(VGG) (very deep architecture) as shown in Fig. 5, where the
proposed method constructs the components of convolutional
layers by considering convolutional and max-pooling layers from
the basic VGG [15]. Then the components are used to extract
features that represent the input image. Before passing the input
image to the network, all the images are scaled to a standard size,
namely, 32 × 100 × 3, as shown in Fig. 6. For the input image in
Fig. 7a, the proposed method generates a feature map as shown in
Fig. 7b, from which the sequence of the feature are extracted
Fig. 7 Feature extraction using CNN
given by the convolutional layers. The output of this process is
considered as the input for the recurrent layers. In other words, the a Input license plate images
b Feature maps of CNN forward propagation
proposed method scans the feature map from left to right column
by column to generate feature vectors of a feature sequence.
Therefore, each column of the feature map corresponds to a information and in turn help in obtaining stable sequence recog-
rectangle region of the input image. The proposed method nition than treating each feature independently as defined in (8).
arranges rectangle regions in the order from left to right with LSTM is illustrated in Fig. 8a, which comprises memory cells and
respect columns. three multiplicative gates, namely, the input, output and forget
It is noted that traditional gradient vanishing does not have gates. In general, memory cells are used to store the past context
the ability to extract context to predict the next state [14]. To find information, the input and output cells are used to store the
a solution to this issue, the long-short term memory (LSTM) context for a long period of time. Similarly, a forget cell is used to
has been developed, which is a special type of RNN unit. The clear the context information in the cells. To strengthen the feature
RNNs are special neural networks, which exploit past contextual extraction and selection, the proposed method combines both

CAAI Trans. Intell. Technol., pp. 1–7

4 This is an open access article published by the IET, Chinese Association for Artificial Intelligence and
Chongqing University of Technology under the Creative Commons Attribution-NoDerivs License
(http://creativecommons.org/licenses/by-nd/3.0/)
 Table 1 Summary of proposed CNN architecture
Layer type Parameter

Convolution Kernel size: 3 × 3, Stride: 1, Padding: ‘SAME’, channel: 64

MaxPooling Kernel size: 2 × 2, Stride: 2, Padding: ‘VALID’
Convolution Kernel size: 3 × 3, Stride: 1, Padding: ‘SAME’, channel: 128
MaxPooling Kernel size: 2 × 2, Stride: 2, Padding: ‘VALID’
Convolution Kernel size: 3 × 3, Stride: 1, Padding: ‘SAME’, channel: 256
Convolution Kernel size: 3 × 3, Stride: 1, Padding: ‘SAME’, channel: 256
MaxPooling Kernel size: 2 × 1, Stride: 2 × 1, Padding: ‘VALID’
Convolution Kernel size: 3 × 3, Stride: 1, Padding: ‘SAME’, channel: 512
BatchNormalization
Convolution Kernel size: 3 × 3, Stride: 1, Padding: ‘SAME’, channel: 512
BatchNormalization
MaxPooling Kernel size: 2 × 1, Stride: 2 × 1, Padding: ‘VALID’
Convolution Kernel size: 2 × 2, Stride: 2 × 1, Padding: ‘SAME’, channel: 512

The padding type ‘Valid’ means no padding, ‘SAME’ means that the
output feature map has the same spatial dimensions as the input feature
map. Zero padding is introduced to make the shapes match as needed,
equally on every side of the input map.

The configuration information of the proposed architecture is

summarised in Table 1.

4 Experimental results

Since the proposed method consists of two steps, we conduct experi-

ments on classification and recognition. For evaluating the proposed
method, we consider live data given by the MIMOS company
funded by Malaysian Government [16], which comprises 874
private images and 961 public images. This dataset includes
images affected multiple adverse factors such as low resolution,
blur and severe illumination effect, images with different back-
ground colour, and images affected by head light, speed of vehicles
and so on. We also consider one more benchmark dataset, namely,
UCSD [17], which consist of 18,270 images. This dataset includes
images captured by different angles, distances and multiple cars in
one image. In total, there are 20,105 images for experimentation in
Fig. 8 Bi-directional LSTM structure this work. Note: since MIMOS dataset contains two different types
a LSTM structure unit
images, namely, private which has a dark background, and the
b LSTM forward and backward directions public which has a white background as shown in Figs. 9a and b,
respectively, where one can see a different colour for the back-
ground. On the other hand, UCSD images have a white background
forward and backward directional LSTM as shown in Fig. 8b like public images in MIMOS as shown in Fig. 9c, where we can see
as defined in (9), which considers the past and future context. The images suffer from poor quality and degradations. Therefore, we use
proposed method uses the soft-max layer for transforming the the MIMOS dataset for classification and show classification
BLSTM states into probability distribution of each character class. is useful for improving recognition performance. However, we
Further, to convert the sequence of the probability, the consider UCSD as a benchmark dataset to show that the proposed
proposed method uses a CTC, which does not require character recognition is effective and generic in this work.
segmentation. This process decodes the predicted probability To measure the performance of the classification step, we consider
as output labels [15]. For instance, the proposed method decodes classification rate with a confusion matrix for experiments on
the label sequence, -M-AA-C-33-4-566-7- as MAC34567 by classification. Since there is no ground truth for the dataset, we
combining two nearest characters which are the same, and deletes count manually to calculate measures. Similarly, to measure the per-
the empty reference ‘-’. Finally, the proposed method chooses the formance of recognition, we consider the recognition rate. In addition,
one with the greatest probability for recognition as defined in (10) to validate the classification, we calculate recognition rates before and
after classification. Before classification considers images of two
classes as the input for calculating recognition rates using different
ht = RNN(xt , ht−1 ) (8) methods. After classification includes images of individual classes
for calculating recognition rates using different methods. Besides,
where ht denotes the memory state, xt denotes the current input and the recognition methods are tuned according to the complexity of
ht−1 denotes the previous state the classes after classification such that the same method can score
better recognition rate compared to before classification.
To show the proposed classification is superior to the existing
hft = LSTMf (xt , hft−1 ) methods, we implement two latest classification methods. The first
(9) one is Xu et al.’s method [5], which explores the uniform colour
hbt = LSTMb (xt , hbt+1 ) of text components for the classification of captions and scene
texts in video. The second one is Roy et al.’s method [7], which
where f refers the forward LSTM, b refers the backward LSTM
proposes tampered features for separating captions and scene texts
in video. The main reason to choose these two existing methods
label = arg max P(l|s) (10) for the comparative study is that both the methods have the same
l objective as the proposed method. The methods consider scene

CAAI Trans. Intell. Technol., pp. 1–7

This is an open access article published by the IET, Chinese Association for Artificial Intelligence and 5
Chongqing University of Technology under the Creative Commons Attribution-NoDerivs License
(http://creativecommons.org/licenses/by-nd/3.0/)
Fig. 9 Sample images of the different database
a Normal plate images from MIMOS
b Taxi plate images from MIMOS
c Sample images from USCD

texts are unpredictable, which suffer from distortions affected Fig. 10 Sample successful and unsuccessful images of the proposed method
by multiple causes as public images in this work. Similarly, the a Sample successful images
methods consider caption texts have good clarity and contrast, b Sample unsuccessful images
which is the same as private images compared to public images.
For recognition, we implement one state-of-the-art method [2]
which proposes stroklets for recognition of texts in natural scene Table 2 Confusion matrix of the proposed method and existing
images. Since license plate recognition is almost the same as methods (%)
natural scene text recognition, we use the above existing methods Proposed method Xu et al. [5] Roy et al. [7]
for comparative study in this work. In addition, according to the
method, it is robust to the poses caused by distortion and generic. Private Public Private Public Private Public
The method [2] uses a random forest classifier for recognition. In
order to test the features extracted by the method, we pass the private 77.54 22.46 57.6 42.4 53.5 46.5
public 19.78 80.22 33.69 66.31 31.98 68.02
same features to CNN for recognition, which is considered as one
more existing method for comparative study. We also use the
Bold values indicate, highest results achieved by the respective methods.
system which is available online [3] for license plate recognition.
This system explores deep learning as the proposed method for
recognition. In summary, we choose one state-of-the-art method
from natural scene text and one more from license plat recognition
to show that existing natural scene text and license plate
recognition methods may not work well for license plate images
affected by multiple adverse factors.

4.1 Evaluating classification

Sample successful and unsuccessful classification images of the pro-

posed classification are shown in Figs. 10a and b, respectively.
Fig. 10 shows that the proposed classification works well for
images with blur, low contrast and illumination effects. However,
the proposed classification fails to classify images that have too
low resolution and poor quality. Therefore, there is a scope for
improvement and extension of the proposed classification in the
future. Quantitative results of the proposed classification and existing
methods are reported in Table 2, where it is noted that the proposed
classification gives better results than the existing methods. The
reason for poor results of the existing methods is that the existing
methods depend on character shapes, while the proposed classifica-
tion depends on the distribution of foreground and background
pixels. For instance, tampered features proposed by Roy et al. [7]
Fig. 11 Recognition results of the proposed and existing methods on
exist only for caption text but not license plate images.
different databases

4.2 Evaluating recognition

extracted strokelet features are not robust to segment characters
For recognition experiments, we split the dataset into two training from such images. When we use strokelet + CNN, the recognition
and testing samples. For all the experiments, we consider 70% for results are better than the stroklet alone method. However, if the
training, 15% for validating and 15% for testing of the datasets. character segmentation step misses characters, recognition by CNN
Qualitative results of the proposed recognition and existing does not make difference in recognition. Similarly, the HLPR
methods are shown in Fig. 11, where it can be seen that the system is better than strokelet and strokelet + CNN but lower than
proposed method recognises all the input images of three different the proposed method. When we look at the failure results of the
databases accurately. However, exiting methods do not work for HLPR system [3], the system is not robust to poor quality and
all the images. Fig. 11 shows that the strokelet based method [2] degraded images. On the other hand, the proposed recognition is
misses characters for several input images especially those affected better than existing methods in terms of robustness. The main
by blur and suffer from poor quality. This is due to that the reason for good results is that the way we use CNN for feature

CAAI Trans. Intell. Technol., pp. 1–7

6 This is an open access article published by the IET, Chinese Association for Artificial Intelligence and
Chongqing University of Technology under the Creative Commons Attribution-NoDerivs License
(http://creativecommons.org/licenses/by-nd/3.0/)
Table 3 Recognition rate of the proposed and existing methods for a proposed method explores foreground and background separation
prior and after classification and then clustering in a new way. For recognition, we combine the
Methods UCSD Before classification After classification merits of CNN and RNN to handle the issues of poor quality,
complex background, blur and noise. Experimental results on
Private + public Private Public classification show that the proposed classification is better than
the existing methods in terms of classification rate. Similarly,
proposed 86.37 71.23 77.03 90.51 recognition experiments on MIMOS dataset and benchmark UCSD
Strokelet [2] 23.76 22.72 34.42 24.90 dataset show that the proposed recognition is better than existing
Strokelet + CNN 34.39 34.78 55.31 33.64
HLPR [3] 83.65 69.53 79.34 85.32 methods. In addition, the recognition performance of different
recognition methods including the proposed method before and
Bold values indicate, highest results achieved by the respective methods. after classification shows that classification is useful for achieving
high recognition rate for a complex dataset rather than developing
a new method. As discussed in Section 4, since there are
limitations to the proposed classification and recognition, we have
planned to overcome those limitations in the near future.

6 Acknowledgment

This research work was supported by the Faculty of Computer

Science and Information Technology, the University of Malaya
under a special allocation of Post Graduate Funding for the
RP036B-15AET project. The work described in this paper was
supported by the Natural Science Foundation of China under grant
no. 61672273, and the Science Foundation for Distinguished
Young Scholars of Jiangsu under grant no. BK20160021.
Fig. 12 Limitation of the proposed method

extraction and BLSTM makes sense for tackling the challenges 7 References
posed by adverse factors. This is the advantage of the proposed
recognition compared to existing methods. [1] Du, S., Ibrahim, M., Shehata, M., et al.: ‘Automatic license plate recognition
(ALPR): a state-of-the-art review’, IEEE Trans. Circuits Syst. Video Technol.,
Quantitative results of the proposed recognition and existing 2013, 1, pp. 311–325
methods are reported in Table 3. It is observed from Table 3 that [2] Bai, X., Cong, Y., Wenyu, L.: ‘Strokelets: a learned multi-scale mid-level
the proposed recognition scores the best for UCSD and before-after representation for scene text recognition’, IEEE Trans. Image Process., 2016,
classification of MIMOS dataset compared to the three existing 25, pp. 2789–2802
[3] ‘High Accuracy Chinese Plate Recognition Framework’, https://github.com/
methods. The HLPR system is the second best compared for all the zeusees/HyperLPR, Accessed 18 May 2018
experiments. Interestingly, when we look at the results of before clas- [4] Raghunandan, K.S., Shivakumara, P., Kumar, G.H., et al.: ‘New sharpness
sification and after classification, all the methods including the pro- features for image type classification based on textual information’.
posed method report low recognition rate prior to classification and Proc. Document Analysis Systems (DAS), Greece, 2016, pp. 204–209
[5] Xu, J., Shivakumara, P., Lu, T., et al.: ‘Graphics and scene text classification in
high recognition rate after classification. The difference between video’. Proc. Int. Conf. on Pattern Recognition (ICPR), Sweden, 2014,
before and after is significant. This is due to the advantage of classi- pp. 4714–4719
fication MIMOS data into private and public, which helps us to [6] Shivakumara, P., Kumar, N.V., Guru, D.S., et al.: ‘Separation of graphics
reduce the complexity. Because of classification, we can tune or (superimposed) and scene text in video’. Proc. Document Analysis Systems
(DAS), France, 2014, pp. 344–348
modify parameters for tackling the issues with the dataset. In this [7] Roy, S., Shivakumara, P., Pal, U., et al.: ‘New tampered features for scene and
experiment, we tune the key parameters of CNN and BLSTM based caption text classification in video frame’. Proc. Int. Conf. on Frontiers in
on training samples with respect classes. Therefore, we can conclude Handwriting Recognition (ICFHR), China, 2016, pp. 36–41
that the recognition performance improves significantly when we [8] Yang, Y., Li, D., Duan, Z.: ‘Chinese vehicle license plate recognition using kernel
based extreme learning machine with deep convolutional features’, IET Intell.
classify complex data into sub-classes according to complexity. In Transp. Syst., 2018, 12, pp. 213–219
other words, rather than developing a new method, this is one way [9] Abedin, M.Z., Nath, A.C., Dhar, P., et al.: ‘License plate recognition system
of achieving a high recognition rate for complex data. It is also based on contour properties and deep learning model’. Proc. Region 10
observed from Table 3 that the proposed recognition scores the best Humanitarian Technology Conf. (R10-HTC), Bangladesh, 2017, pp. 590–593
[10] Polishetty, R., Roopaei, M., Rad, P: ‘A next generation secure cloud based deep
recognition rate for the UCSD dataset, which shows the method can learning license plate recognition for smart cities’. Proc. Int. Conf. on Machine
be used for any other dataset irrespective of scripts. The main Learning and Applications (ICMLA), USA, 2016, pp. 286–294
reason for the poor results of the existing methods is the same as dis- [11] Montazzolli, S., Jung, C.: ‘Real time Brazilian license plate detection and
cussed earlier. recognition using deep convolutional neural networks’. Proc. SIBGRAPI on
Conf. on Graphics, Patterns and Images, Brazil, 2017, pp. 55–62
When the images are oriented in different directions, the proposed [12] Bulan, O., Kozitsky, V., Ramesh, P., et al.: ‘Segmentation and annotation free
recognition method does not work well as shown sample results in license plate recognition with deep localization and failure identification’, IEEE
Fig. 12. This is not a problem of feature extraction but it is the Trans. Intell. Transp. Syst., 2017, 18, pp. 2351–2363
limitation of CTC, which gets confusing when an image is rotated [13] Selmi, Z., Halima, M.B., Alimi, A.M.: ‘Deep learning system for automatic
license plate detection and recognition’. Proc. Int. Conf. on Document Analysis
in different directions. However, it is noted that in the case of and Recognition (ICDAR), Japan, 2017, pp. 1132–1137
license plate images, it is seldom to have different orientations like [14] Li, H., Wang, P., You, M., et al.: ‘Reading car license plates using deep neural
texts in natural scene images. Therefore, this issue may not be a networks’, Image Vis. Comput., 2018, 72, pp. 14–23
major drawback of the proposed recognition method. [15] Shi, B., Xiang, B., Cong, Y.: ‘An end-to-end trainable neural network for
image-based sequence recognition and its application to scene text
recognition’, IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39, (11),
pp. 2298–2304
5 Conclusion and future work [16] Shivakumara, P., Konwer, A., Bhowmick, A., et al.: ‘A new GVF arrow pattern
for character segmentation from double line license plate images’. Proc. Asian
Conf. on Pattern Recognition (ACPR), China, 2017, pp. 782–787
In this work, we have proposed the combination of CNN and RNN [17] Zamberletti, A., Gallo, I., Noce, L.: ‘Augmented text character proposal
for recognising license plate images through classification. For and convolutional neural networks for text spotting from scene images’.
the classification of private and public number plate images, the Proc. Asian Conf. on Pattern Recognition (ACPR), Malaysia, 2015

CAAI Trans. Intell. Technol., pp. 1–7

This is an open access article published by the IET, Chinese Association for Artificial Intelligence and 7
Chongqing University of Technology under the Creative Commons Attribution-NoDerivs License
(http://creativecommons.org/licenses/by-nd/3.0/)
View publication stats