Fully Automatic LPR Method Using Haar

Cascade for Real Mercosur License Plates

Cyro M. G. Sabóia, Adriell G. Marques, Luı́s Fabrı́cio de Freitas Souza, Solon

Alves Peixoto, Matheus A. dos Santos, Antônio Carlos da Silva Barros, Paulo A.
L. Rego, and Pedro Pedrosa Rebouças Filho

Laboratório de Processamento de Imagens, Sinais e Computação Aplicada,

Instituto Federal do Ceará, Universidade Federal do Ceará - Brazil
[email protected],
WWW home page: http://lapisco.ifce.edu.br

Abstract. The growing increase in traffic and road monitoring technolo-

gies brings new challenges and possibilities for using character detection
and recognition technologies to improve traffic management and road
safety. This work proposes a study between a model trained with synthetic
license plate images and a model trained with real license plate images.
A new LPR-UFC database is also presented to be made available via a
request containing 2.686 images of vehicles with Mercosur plates [1]. To-
gether with the perspective adjustment, the model result obtained a gain
of 97.00% of accuracy for detection and 88.48% accuracy for recognition.

Keywords: License Plate Detection, License Plate Recognition, Perspec-

tive Adjustment, Haar Cascade, Tesseract.

1 Introduction

With the advancement of traffic technologies, intelligent systems of detection

and recognition of traffic images become more and more critical to help the
management of highways, improving the collection of information and mapping of
traffic conditions [2]. Technologies for detection and recognition of license plates,
whether of roads or vehicles, are still objects of relevant studies in computer
vision, whether for road safety applications, parking, tolls, stolen vehicle tracking,
border control, speed, etc. [3–6].
The License Plate Recognition (LPR) techniques maturation in computer
vision allowed advances in discussions about license plate detection and the
recognition of digits in images provided by video cameras [7], whether embedded
in official vehicles or at specific points on the highways, allowing the development
of more specific applications in real-time [8].
One of the known computer vision techniques for object detection is the Haar
Cascade [9], which consists of a list of steps that label the specific region of the
image as positive or negative. Positive images have the object you want to find,
and negative images don’t have the object you want to find. [10].
2 Cyro M. G. Sabóia et al.

Fig. 1. It presents the steps of the model of this study. Stage 1 - Dataset Input Imgeries,
Stage 2 - Trained Haar Cascade Model, Stage 3 - License Plate Segmentation, Stage 4 -
License Plate Perspective Adjustment, Stage 5 - License Plate pre-processing, Stage
6 - License Plate Recognition, Stage 7 - Recognition Text Result, Stage 8 - Heuristic
Algorithm, Stage 9 - License Plate Text Result After Heuristic Algorithm.

For character recognition, the computer vision technique Optical Character

Recognition (OCR) is applied to the images of the detected license plates. OCR
retrieves the image objects in the form of attributes and expresses it in textual
information [11].
With the increasing demand for LPR applications and their challenges, this
work proposes an approach of automatic detection of plates in the Haar cascade,
followed by character recognition using Tesseract OCR [12–14]. A new LPR-
UFC database containing 2.686 images will also be available to the community
upon request. This work seeks to broaden the discussions on Haar Cascade as a
computer vision tool for plate detection, Tesseract OCR for character recognition,
and a new database containing only Brazilian plates from the new Mercosur
model.
As main contributions of this study, the different relevant contexts are ad-
dressed:
– Mercosur license plate model detection in real imagerys.
– Character recognition of Mercosur license plate model in real imageries.
– A comparative study between a model trained with images of synthetic license
plates and a model trained with images of real license plates.
– Detection and recognition of Mercosur license plates in adjusted perspective
angles using Tesseract OCR.
– Increase of the LRP-UFC dataset with more 1.586 real images of vehicles
with the new Mercosur license plate model.

2 Related Works
Cyro M. G. Sabóia et al. [15] performed an LPR study on synthetically generated
license plates. As the Mercosur plate model was recently released, a database
 ALPR Using Haar Cascade and Tesseract OCR 3

with real imageries was not yet in the public database. From the synthetically
generated license plate imageries dataset, the study proposed using Haar Cascade
trained from a set of synthetically generated positive and negative images that
did not contain license plates, to perform LPR on the Brazilian license plate in
the new format of the Mercosur. The study achieved satisfactory results for the
license plate detection problem.
the work of Luı́s Fabrı́cio de F. Souza et al. [8] proposed to apply the model
trained on synthetic plates by [15] in a dataset of real LPR-UFC imageries.
In the research, perspective adjustment techniques were used, which proved
significant gains in character recognition. The study achieved 90.00% in license
plate detection and 83.01% in character recognition.
Al Awaimri et al. [16] developed an Omani license plate detection and recog-
nition system using different methodologies, such as optical character recognition,
convolutional neural network, or deep neural network. Several works were carried
out an analytical study to understand which algorithms are more suitable, and
finally, a practical study on real boards. As a result, the work achieved 71.5% for
license plate extraction and 96.00% to 99.00% accuracy in character recognition.
The deep-learned detection approach system proposed by [17] is divided into
image acquisition, license plate detection, and optical character recognition. The
architecture used is the Mobilenet V1 based on Single Shot Detector (SSD),
allowing the processing of images with accuracy superior to 95.00%.
Considering the different models of plates existing in the world [18] proposed
a precise approach applicable to license plates from other countries, using tiny
YOLOV3 for license plate detection, YOLOv3-SPP for unified character recog-
nition, and, In order, it is proposed layout detection algorithm to extract the
correct sequence from the LP number. According to the authors, the results
showed that the results presented surpassed previous research works.

3 Materials

This section provides information about the dataset used in this article. One
thousand five hundred eighty-six images increased the LPR-UFC dataset to
train the proposed method, now having 2.686 imageries of vehicles with the new
Mercosur license plate model. We took one hundred imageries for testing, and the
rest of the dataset was resized to 7 different sizes, as shown in Table 2, totaling
17.962 positive images and 35.658 negative images.
Also, as a contribution to this work, the LPR-UFC dataset was increased
by 1586 more images from the LPR-UFC dataset provided by [8], totaling
2.686 images. The images from the new LPR-UFC dataset were collected in
Fortaleza city, Brazil, from different angles and cameras. (Models: Intelbras and
Hikvision) configured with 1920 x 1080px resolution, 8192Kbps bit rate, H .265
encoding and 25fps refresh rate. The dataset is available upon request by email:
[email protected], subject to citation of this study.
4 Cyro M. G. Sabóia et al.

4 Proposed Methodology

Fig. 2. Proposed methodology flowchart.

This section will present the methodology used for the LPR process applied
to this study. In work presented by [15], a Haar Cascade model trained with
synthetic imageries is applied to a dataset with synthetic imageries. In [8], the
same model trained on synthetic imageries is applied to a set of real license plate
imageries. The method proposed in this work is to train the Haar Cascade model
with real imageries and apply it to the real license plate images dataset. Research
Methodology Divided into eight stages, defined in Stage 1 - Database with real
license plates images in the new Mercosur format, Stage 2 - Haar Cascade model
training, Stage 3 - Segmentation, Stage 4 - Perspective adjustment, Stage 5 -
pre-processing, Stage 6 - Optical Character Recognition OCR, Stage 7 - Text
recognition result, Stage 8 - Heuristic algorithm, Stage 9 - Resulting text.
Stage 1 - Database with real imageries of license plates in the new
Mercosur format - In this step, a database is built containing vehicles with
plates of the new Mercosul Figure 3 format, both with frontal and rear perspec-
tives. The database has 2686 images, 2.586 used in training, and 100 separated
for testing.

Fig. 3. Real dataset images sample from different vehicles.

 ALPR Using Haar Cascade and Tesseract OCR 5

Stage 2 - Haar Cascade model training - Model training was done with
2586 imageries separated for training. The Cascade Trainer GUI tool (Version
3.3.1) [19] software was used for this. The imageries of the plates applied to the
tests were cut from the original image containing the entire vehicle. Then only
the license plate imageries were resized to seven different sizes according to Table
2, forming the basis of positive images, which contain the object to be detected.
The same dataset used in [15] was used for negative imageries.
Stage 3 - Segmentation - In this step, the 100 images of vehicles with
Mercosur plates separated for testing are passed through the trained model,
resulting in new images containing only the cutout of the plate Figure 4.

Fig. 4. Cropped license plate sample in different license plates.

Stage 4 - Perspeective Adjustment - After the segmentation stage, the

license plate cropped image is passed to the perspective adjustment algorithm,
which corrects the skew, leaving the plate correctly aligned Figure 5.

Fig. 5. Perspective Adjustment sample in different license plates. Compare with Fig. 4

Stage 5 - Preprocessing - In the pre-processing stage, the license plate

imageries go through a series of functionalities made available by the OPENCV
library, where the image is converted to gray scale, the application of filters for
noise smoothing, binarization and morphological operations to close gaps between
digits Figure 6.

Fig. 6. Pre-processing: Smoothed, binarized image and morphological operations applied

in different license plates.

Stage 6 - Optical OCR Character Recognition - The result of the

pre-processing of the image is sent to the optical character recognition tool
Tesseract-OCR, in its python version called Pytesseract.
Stage 7 - Textual recognition result - In this step the Tesseract returns
the recognized text of the Figure 7 image. It is important to understand that the
6 Cyro M. G. Sabóia et al.

sequence of plates in the new Mercosur format is composed of three letters, one
digit, one letter, and two more digits, thus totaling seven characters per plate.
Note that the tesseract can confuse letters, digits, letters to digits, and vice versa.

Fig. 7. Character recognition result in different license plates.

Stage 8 - heuristic algorithm - The license plate format in the new Merco-
sur model follows the standard with seven digits in the following sequence: letter,
letter, letter, number, letter, number, number. Sometimes character recognition
can recognize a letter when a number is expected, such as replacing the letter
O with the Number 0. To optimize the results is used a heuristic algorithm to
change characters when a letter is expected, and a digit appears, and vice versa.
The algorithm logic can be found in Table 1.

Table 1. Replacing Heuristic.

Letter Q D O I Z A S G T B
⇓ ⇓ m m m m m m m m
Digit 0 0 0 1 2 4 5 6 7 8

Stage 9 - Result text - As a result of the heuristic algorithm, a new

resulting text is reported, correcting flaws and optimizing the recognition results
fig: heuristicr esult.

Fig. 8. Heuristic result in different license plates.

5 Results and Discussion

In this section, we will describe the results and discussion of the experiments
carried out in the study. In work proposed by [15], a model using the Haar Cascade
 ALPR Using Haar Cascade and Tesseract OCR 7

method was trained using synthetically generated images. In work presented

by [8], this same model trained with synthetic images was applied in the tests of
real imageries. The proposed method of this work is to train a new model with
real imageries and use it again to the training images of the previous models. We
will address the results in two subsections: License Plate Detection and License
Plate recognition.
The work was developed and tested on a computer with the following config-
urations: Intel Core i7 processor with 2.9GHz, 8 GB RAM, Ubuntu 16.04 LTS
operating system.

5.1 License Plate Haar-Cascade Trainning

OpenCV provides us with tools for training and augmenting samples from the
positive image pool. In this work, we use the Cascade Trainer GUI tool (Version
3.3.1). The training consisted of 16 stages, a sample width of 40, and a sample
height of 13. For detection, scaleFactor 1.1, minNeighbors 1, minSize(1,1), and
maxSize(300,300). Table 2 is defined the amount of positive and negative images
used in training.
For training the model, a dataset of real imageries was created in Section. 3
Materials. The 2.586 images used in training were resized to seven different sizes,
resulting in 17.962 Tab positive images. 2. For the negative images, the same
dataset with 35.658 negative images described in [15] was used.

5.2 License Plate Detection

For a better understanding, it is important to highlight that the detection model

trained in this experiment was applied only to the Real [8] test base.
In tests performed without the perspective adjustment for the Real database,
the model trained in this work presented a result of 91.00% of accuracy. There
was a gain of 1.00% in the model trained on the Real base applied to the Real
test database
As for the tests performed with perspective adjustment, the model trained
in this work presented a result of 97.00% of accuracy. There was also a gain,
with an increase of 7.00% in the model trained on the real base applied to the
Real test database. The gain was higher because the Real database had a greater
amount of inclined plates. The synthetic database, precisely because a computer
creates it, has a more homogeneous alignment and fewer obstacles for detection.
Both results can be seen in Table 3.

5.3 License Plate Recognition

In Table 4 we present the results of character recognition. For a better under-

standing, it is important to note that the recognition was applied only to the
same test base used in [8]. Both the results with perspective adjustment and
without perspective adjustment showed a gain from the results presented by [8].
8 Cyro M. G. Sabóia et al.

For character recognition without perspective adjustment, we got a result of

86.05% against 83.01%, showing an increase of 3.04%. For character recognition
with perspective adjustment, the result was 88.48% against 83.13%, showing an
increase of 5.35% both compared to the results of [8].

Table 2. License Plate Haar-Cascade Trainning.

Positives Negatives Images Size

17.962 35.568 60x16 80x22 100x27 120x32 140x38 160x44 180x49

Table 3. License plate detection models and respective results.

No Perspective Adjustment Method Train test ACC (%)

Cyro M. G. Sabóia [15] Syntectic Dataset Syntectic Dataset 83.82%
Luı́s Fabrı́cio de F. Souza [8] Syntectic Dataset Real Dataset 90.00%
Proposed Method Real Dataset Real Dataset 91.00%

With Perspective Adjustment Method Train test ACC (%)

Cyro M. G. Sabóia [15] Syntectic Dataset Syntectic Dataset 90.00%
Luı́s Fabrı́cio de F. Souza [8] Syntectic Dataset Real Dataset 90.00%
Proposed Method Real Dataset Real Dataset 97.00%

Table 4. Character Recognition models Comparison using Tesseract OCR - Real and
synthetic images.

No Perspective Adjustment Method Experiments ACC

Cyro M. G. Sabóia [15] Syntectic Dataset 95.72%
Luı́s Fabrı́cio de F. Souza [8] Real Dataset 83.01%
Proposed Method Real Dataset 86.05%

With Perspective Adjustment Method Experiments ACC

Cyro M. G. Sabóia [15] Syntectic Dataset 95.72%
Luı́s Fabrı́cio de F. Souza [8] Real Dataset 83.13%
Proposed Method Real Dataset 88.48%

The proposed model, compared to the methods in the literature [15] and
[8], brought superior and satisfactory results. Based on the methods found for
the same problem, the perspective adjustment method made the license plate
detection process more robust, helping identify characters in real imageries. The
performance is measured by the accuracy based on the amount of identification
of license plates and accuracy in identifying the characters. In the processing
 ALPR Using Haar Cascade and Tesseract OCR 9

time of the tests, the proposed model obtained an average time of 22 hundredths
of a second, bringing robustness to the process.

6 Conclusion and Future Work

In this study, a complete method of license plate detection and character recogni-
tion was proposed in an unprecedented database of images of vehicles with license
plates in the new format of Mercosur Figure 1. The imageries are from natural
environments, taken by cameras fixed on busy avenues in Fortaleza, Brazil.
The studied model presented satisfactory results and was consistent with state-
of-the-art, with 91.00% and 97.00% accuracy in detection without perspective
adjustment and perspective adjustment, respectively. For character recognition,
the results were 86.05% and 88.48% accuracy without perspective adjustment
and with perspective adjustment, respectively.
In both cases, there were improvements in the results, showing that Haar
Cascade identifies the specifics of synthetic and real imageries, such as texture,
shading, contrast, etc.
For future work, we’ll use different classification and detection methods, using
the YOLO network, to identify the old Brazilian license plate model and the new
Mercosur license plate model.

Acknowledgement
The authors would like to thank The Ceará State Foundation for the Support of
Scientific and Technological Development (FUNCAP) for the financial support
(grant #6945087/2019).

References
1. G. Silvano, V. Ribeiro, V. Greati, A. Bezerra, I. Silva, P. T. Endo, and T. Lynn,
“Synthetic image generation for training deep learning-based automated license plate
recognition systems on the brazilian mercosur standard,” Design Automation for
Embedded Systems, vol. 25, no. 2, pp. 113–133, 2021.
2. C. Zhang, Y. Tai, Q. Li, T. Jiang, W. Mao, and H. Dong, “License plate recognition
system based on opencv,” in 3D Imaging Technologies—Multi-dimensional Signal
Processing and Deep Learning, L. C. Jain, R. Kountchev, and J. Shi, Eds. Singapore:
Springer Singapore, 2021, pp. 251–256.
3. M. Bensouilah, M. N. Zennir, and M. Taffar, “An alpr system-based deep networks
for the detection and recognition.” in ICPRAM, 2021, pp. 204–211.
4. G. V. Vi and A. A. b. M. Faudzi, “A study on different techniques in alpr system:
The systems performance analysis,” in Recent Trends in Mechatronics Towards
Industry 4.0, A. F. Ab. Nasir, A. N. Ibrahim, I. Ishak, N. Mat Yahya, M. A. Zakaria,
and A. P. P. Abdul Majeed, Eds. Singapore: Springer Singapore, 2022, pp. 617–627.
5. R. Balia, S. Barra, S. Carta, G. Fenu, A. S. Podda, and N. Sansoni, “A deep learning
solution for integrated traffic control through automatic license plate recognition,” in
International Conference on Computational Science and Its Applications. Springer,
2021, pp. 211–226.
10 Cyro M. G. Sabóia et al.

6. P. Sharma, S. Gupta, P. Singh, K. Shejul, and D. Reddy, “Automatic number

plate recognition and parking management,” in 2022 International Conference on
Advances in Computing, Communication and Applied Informatics (ACCAI). IEEE,
2022, pp. 1–8.
7. P. Srikanth and A. Kumar, “Automatic vehicle number plate detection and recog-
nition systems: Survey and implementation,” in Autonomous and Connected Heavy
Vehicle Technology. Elsevier, 2022, pp. 125–139.
8. L. F. de F. Souza, C. M. G. Sabóia, A. G. Marques, J. da Costa Nascimento, A. C.
d. S. B. Matheus A. dos Santos, P. A. L. Rego, and P. P. R. Filho, “New approach
to the detection and recognition of brazilian mercosur plates using haar cascade
and tesseract ocr in real images,” Journal of Information Assurance and Security,
vol. 17, pp. 144–153, 2022.
9. A. Takiddin, M. Shaqfeh, O. Boyaci, E. Serpedin, and M. Stotland, “Gauging facial
abnormality using haar-cascade object detector,” in 2022 44th Annual International
Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE,
2022, pp. 1448–1451.
10. S. D. Yaddanapudi, B. P. Makala, A. Yarlagadda, C. T. Sapram, S. T. Parsa, and
S. Nallamadugu, “Collection of plastic bottles by reverse vending machine using
object detection technique,” Materials Today: Proceedings, 2021.
11. C. Thorat, A. Bhat, P. Sawant, I. Bartakke, and S. Shirsath, “A detailed review on
text extraction using optical character recognition,” in ICT Analysis and Applica-
tions, S. Fong, N. Dey, and A. Joshi, Eds. Singapore: Springer Singapore, 2022,
pp. 719–728.
12. M. Niluckshini and M. Firdhous, “Automatic number plate detection using haar-
cascade algorithm proposed for srilankan context,” in 2022 2nd International
Conference on Advanced Research in Computing (ICARC). IEEE, 2022, pp.
248–253.
13. W. Thumthong, P. Meesud, and P. Jarupunphol, “Automatic detection and recog-
nition of thai vehicle license plate from cctv images,” in 2021 13th International
Conference on Information Technology and Electrical Engineering (ICITEE). IEEE,
2021, pp. 143–146.
14. S. Dome and A. P. Sathe, “Optical charater recognition using tesseract and clas-
sification,” in 2021 International Conference on Emerging Smart Computing and
Informatics (ESCI), 2021, pp. 153–158.
15. C. M. G. Sabóia and P. P. R. Filho, “Brazilian mercosur license plate detection
and recognition using haar cascade and tesseract ocr on synthetic imagery,” pp.
849–858, 2022.
16. M. Al Awaimri, S. Fageeri, A. Moyaid, C. Thron, and A. ALhasanat, Automatic
Number Plate Recognition System for Oman. Cham: Springer International Pub-
lishing, 2022, pp. 155–178.
17. N. Awalgaonkar, P. Bartakke, and R. Chaugule, “Automatic license plate recognition
system using ssd,” in 2021 International Symposium of Asian Control Association
on Intelligent Robotics and Industrial Automation (IRIA), 2021, pp. 394–399.
18. C. Henry, S. Y. Ahn, and S.-W. Lee, “Multinational license plate recognition using
generalized character sequence detection,” IEEE Access, vol. 8, pp. 35 185–35 199,
2020.
19. T. Phase, “Building custom haar-cascade classifier for face detection,” International
Journal of Engineering and Technical Research, vol. 8, 01 2020.