LICENSE PLATE NUMBER RECOGNITION

New heuristics and a comparative study of classiers

C sar Garca-Osorio, Jos -Francsico Dez-Pastor e e
Higher Polytechnic School, University of Burgos, Avda. Cantabra, Burgos, Spain [email protected], [email protected]

Juan J. Rodrguez
Higher Polytechnic School, University of Burgos, Avda. Cantabra, Burgos, Spain [email protected]

Jes s Maudes u
Higher Polytechnic School, University of Burgos, Avda. Cantabra, Burgos, Spain [email protected] Keywords: Abstract: Computer Vision, Digital Image Processing, Optical Character Recognition, Smoothing Spatial Filter. We describe an articial vision system used to recognize the Spanish car license plate numbers in raster images. The algorithm is designed to be independent of the distance from the car to the camera, the size of the plate number, the inclination and the light conditions. In the preprocessing steps, the algorithm takes a raster image as input and gives an ordered list of license plate areas candidates. Only in very rare occasions the second and third candidates are needed; most of the cases the rst candidate is the correct one. During the preprocessing a new lter has been used. This lter is only applied to low saturation areas, getting better results this way. In order to choose the best classier for the classifying stage, a comparative study has been performed using the data mining tool Weka.

INTRODUCTION

introduction to the Spanish licence plate number system rst.

License plate number recognition or license plate recognition (LPR) is one of the most practical application of image processing and pattern recognition techniques. Its most common use is the control of parking areas (Sirithinaphong and Chamnongthai, 1999) and trafc monitoring (Setchell, 1997). In these contexts the distance to the camera, position, inclination and other variables that affect the image are quite well controlled and hence it is possible to take advantage of these restricted working conditions (Emiris and Koulouriotis, 2001). The application presented in this paper is designed to work in not so restricted and structured environments. We want to locate the license plate and recognize its number independently of its size, orientation, position or lightening condition. The only restriction we assume is that the background of the license plate number is white and the characters color is black. This is the common style for the Spanish license plates which we want to recognize. The process of license plate number recognition involve three main steps: i) license plate localization, ii) character segmentation, and iii) character recognition. We have structured the rest of the paper in accordance with these steps. However, we will give an

1.1 Spanish License Plates Formats

The increase in the number of cars has put to the limits the license plate number systems. So, plate number systems have needed to be adapted as the number of cars approached the maximum number these systems could register. In Spain, initially, the plate number were composed by one or two letters to code the province followed by up to six digits. This system lasted till 1971 when the plate numbers in Madrid (Urios-Mond jar, 2007), the capital of Spain, were e approaching the number 999999. In the new system there were four digits and a letter in a rst moment, and later was necessary to use two letters. With the joining of Spain to the European union a new change happened. In the year 2000, and close again to the coding limit of the system, a new change is made. In the new system a plate number is composed by four digits followed by three letters. Besides, the province code disappeared and an European Union logo with

With province code and no letter With province code and letters With province code, letters and European Union logo Current plate number code Table 1: Common Spanish plate number formats

blue background and a E in white1 appeared in the left side of the plate number. As well, whenever a plate number need to be substituted because it is deteriorated, the new plate number will came with the European Union logo, even if the plate number is in the old format with the province code. In Table 1 it is possible to see all these formats2 . These and others facts about Spanish plate numbers can be found in the webpage http://www. geocities.com/amoros29/.

Figure 1: Top left: Original RGB image. Bottom left: After applying traditional Sobel lter. Top right: the image without the low saturation areas. Bottom right: After applying the conditional Sobel lter.

2
2.1

LICENSE PLATE LOCALIZATION

Image preprocessing

The rst stage of the process is to locate the licence plate area in the input photo. In order to do so, we need to emphasize regions of high spatial frequency that correspond to edges. Therefore, we rst use the classic Sobel lter. As the input to this step of the process is an RGB color image, and we know that in the license plate the background is white and the characters are black, we will apply the horizontal Sobel lter to the areas of low saturation. The saturation is obtained from RGB color image using the formula if max{R,G,B}=0, otherwise. (1) All the achromatic colors (grey scale colors) have the same saturation and lower than any chromatic color. Note that the white could be actually gray depending S=
max{R,G,B}min{R,G,B} max{R,G,B}

for Espa a, Spain in Spanish n the Spanish plate number system there are two rows plate numbers too, as well as other special plate numbers, that has not been taken into account in the applications explained in this paper.
2 In

1 E

on illumination conditions. The black and white pixels of the license plate areas will have low saturation. So, we discard all the pixels with high saturation and apply the Sobel lter only to the pixels with low saturation. Experimentally, we found that the best results are obtained when applying the Sobel lter to pixels with saturation lower than 0.35. In Figure 1 we can see the results of using this heuristic. Another heuristic we use here is to discard the 30% top part of the image and the 10% of the bottom part. This areas never have a license plate number (Rodrguez, 1999). Next in the process is the binarization. We obtain a black and white image from the result of the previous transformation. The binarization threshold is chosen to get a black and white image with only 2% of the pixels being white, this way only those pixels with high intensity variation are considered as edges in the license plate. The edges due to background, or without that high intensity variation are discarded. After all the previous preprocessing, the license plate will appear as a sequence of white and black transitions with a lot of white pixels concentrated in the license plate area. In other parts of the image the concentration of white pixels is lower. As the last step of this stage we use an horizontal soften lter. This lter can be thought as a kind of mean or average lter. The idea is to count the number of white pixel in a rectangular area with center at each pixel. Empirically, we found the best results were obtained with a size 3x61. If the number of white pixels is greater than 50 we leave that pixel white. That is, M(x, y) = 1 1 (1=white), if 30 i=30 j=1 M(x + i, y + j) > 50. When we apply this lter to a line with a lot of low concentrated edge pixels, all those pixels are elimi-

Figure 2: Left: Image after binarization. Right: Image after applying the horizontal soften lter.

Figure 4: Top left: Original RGB image. Bottom left: After applying traditional Sobel lter. Top right: After binarization. Bottom right: After rotation.

Note, that we keep all the ten areas not discarded. Later, if in the following stages we realize that the selection of plate area was incorrect we recover the following plate candidate from the ordered list. In the experiment this only happened in few occasions.
Figure 3: Left: Image after dilation/erosion with a window 15 pixels width. Right: Image after dilation/erosion with a window 20 pixels width.

3 CHARACTER SEGMENTATION
After the previous stage, we obtained a license plate image. We need to do further image processing before starting to locate the characters in the plate. We have not tried any new heuristic with these steps. We apply the classic Sobel lter (Gonzalez and Woods, 2002), gradient binarization (Rodrguez, 1999) and Hough transform (Gonzalez and Woods, 2002) to correct the plate inclination if needed (see Figure 4).

nated. On the contrary, when we apply this lter to a line with less edge pixels but high concentrated, the line will became white. One could think that the effect of applying this lter is the same as the dilation/erosion of mathematical morphology (Serra, 1982), but this lter seems to be less sensitive to noise. In Figure 3 we can see the results of dilation/erosion with different window sizes.

3.1 Character Localization

Now with the preprocessed plate we use a novel method to locate the characters in the plate. That is one of the most delicate steps of the process. First, we try to adjust further the license plate and we eliminate from the border of the image all the rows and columns with only white or only black pixels. Second, we apply a coarse method to discard those blobs of black pixels which are obviously consequence of noise and are not really characters. This process must be conservative, because we do not want to discard correct characters. We have follow several heuristics: We discard all the areas whose width is greater than 1/8 of the width of the area we have identied as a license plate candidate. As the license plate will have a minimum of seven characters, an area with bigger width will be for sure not a character. We discard as well the areas which are taller than 0.7 the height of the plate. We use the color information to eliminate the areas with at least 50% of high saturation pixels. This areas correspond to the blue European Union

2.2

Selection of plate candidate

After applying the last step of the previous stage, we have a black image with one or more white areas candidate to be a license plate. However, only one of those areas corresponds to the real license plate, while the others are consequence of noise or other objects in the background. Now, we need an algorithm to determine which one of these white areas is the real plate. We have used several heuristics. First, we only keep the ten biggest areas, this way we eliminate most of the areas due to noise. We then order those ten plate candidates according to the following criteria ratio of width to heights greater than two, since a license plate is wider than taller. distance to the center of the image, the areas far from the center are unlikely to be plates. For two areas with similar characteristics we choose the one with lower positions, we do this because we have found that some times the area due to the edges in the car radiator will give end as a false positive plate.

Figure 5: License plate before and after eliminating the black areas that does not correspond with characters.

logo. This have the effect of eliminate other chromatic areas in case our location of the license plate would have not adjust close enough to the plate. If the width of the widest area is lower than 1/20 of the width of the area we have identify as the license plate, we have found a case of false positive. We need to start the process with the next candidate from the list of plate candidates. Next, we apply a more rened strategy. A character will be a blob of connected black pixels (or several, if the characters parts have been disconnected as a consequence of the previous preprocessing) fully inside of a rectangular area of dimensions calculate as a percentage of the dimensions of the license plate area. We align in turn the top left corner of this rectangle with the top left corner of all the blobs, starting with the one in the very top left. Be 1 the blob we have used to align the rectangle. It will not be the case that 1 does not fully x inside the rectangle, because all that big blobs have been discarded with the previous coarse method. Now, if there is piece of another blob 2 that is partially inside the rectangle, but not fully inside, we can discard 1 . It will not be a character or part of a character, 1 is probably noise. If there are not others blobs partially inside the rectangle, and there is another blob 2 fully inside the rectangle, we can consider both 1 and 2 as parts of the same character. This method is able to group pieces of characters that have been disconnected after applying the lters, and at the same time discards any remaining noise. In Figure 6 we can see an example of the method. The coarse method was not able to eliminate all the noise. The plate border is broken and was not big enough as to be discarded. Besides the U appears broken in two pieces. In the proposed method the rectangle will be aligned with the top left pixel of the rst piece, and as the other piece is fully inside the rectangle both pieces are identied as part of the same character. On the contrary, the small blob due to the number plate screw is not consider part of the character because when we align the rectangle with its top left pixel, the six is only partially inside the rectangle. The same happens in the right side, when we align the rectangle with a border, and the Z is only partially inside the rectangle. We have found this method slightly better than the

Figure 6: Character segmentation

traditional horizontal projection method for character segmentation and less sensible to broken characters. Still, in same rare circunstantes we could group some scattered points and identify them as parts of characters. But this cases could be easily managed using as criteria the number of black pixels.

3.2 Character Normalization

Before using a classier to identify the characters we need to normalize them. There are three reasons for normalization: We can reduce the number or attributes and that will make the learning of the classier faster. We will have the same number of attributes for all the characters because they will have the same dimensions. We will improve the efciency of the classier as the input is more similar to the others instances in its class, and more different from the instances in other classes. The normalization steps are: Center the character. Resize it to 16 16 keeping its ratio width/height (this is specially important for characters like I).

4 CHARACTER RECOGNITION
In this last section, we analyze and compare which set of characteristics and classiers give the best classication results. To do this study we use Weka (Witten and Frank, 2002), a data mining tool written in Java (http://www.cs.waikato.ac.nz/ml/weka/). We evaluate the performance of seven different classiers and four different sets of characteristics using 2070 characters from about 700 licence plates. We use 1010-fold cross validation, in each of the experiments 90% of the instances were used for training and 10% for validation, and this was repeated

another nine times, with different 10% of instances each. An important characteristic we could extract from the character image is the number of holes in the characters. That increase the classication accuracy for characters like P and F or B, 8, 6 and 9. The sets of characteristics use in this study were: a. In this data set we represent each of the 2070 characters using the whole 16 16 = 256 pixels of the character image together with the number of holes: 257 attributes in total. This is basically the character without any characteristic extraction process. b. We use the horizontal and vertical projections of the character. This gives 16 characteristics from the horizontal projections, 16 characteristics from the vertical projections, plus the number of holes: 33 characteristics. c. In this data set each character is represented by its main four Kirsch compass lters, horizontal, vertical, and both diagonals. The results are down sampled to a 4 4 image, and the original image is down samples as well. This gives 5 16 = 80 characteristics, plus the additional one for the number of holes: 81 characteristics. d. We use a overlapped down sample of the original image. Each 4 4 block of pixels gives a value, then this mask is moved two pixels right, and so till the right side, then the mask is lower two pixels and the process repeated from the left. We obtain 49 attributes, plus the number of holes: 50 characteristics. Regarding the classiers, we tried the following: 1. functions.MultilayerPerceptron, with parameters: GUI=false, autoBuild=true, Debug=false, Decay=false, HidenLayers=(number of attributes + number of classes)/2, LearningRate=0.3, momentum=0.2, nominalToBinaryFilter=true, normalizeAtributes=true, normalizeNumericClass=true, RandomSeed=0, trainingTime=500, validationSetSize=0 and validationThreshold=20. Neural Networks has been used as classiers in (Draghici, 1997; Nijhuis et al., 1995) 2. bayes.NaiveBayesUpdateable, with options: debug=false, useKernelStimator=false and useSupervisedDiscretization=false. 3. functions.SMO, Platts sequential minimal optimization algorithms for training Support Vector Machines (Platt, 1999; Zheng and He, 2006; Chengwen et al., 2006), with options: buildLogisticModels=false, C=1.0, checksTurnedOff=false, Epsilon=1.0E-12, lterType=Normalize training

1 2 3 4 5 6 7

a. 99.10 96.42 99.19 97.86 97.56 95.46 94.04

b. 97.88 89.44 96.85 96.38 95.39 92.92 89.17

c. 98.15 89.86 98.01 96.45 96.23 93.87 91.04

d. 99.03 93.65 99.09 97.74 96.81 95.94 93.66

Table 2: Results comparison

data, Kernel=PolyKernel, RamdomSeed=1 and toleranceParameter=0.0010. 4. lazy.IBk, with no distance weigthing and options: KNN=1, crossValidate=false, Debug=false, MeanSquared=false, NearestNeighbourSearchAlgoritm=LinearNN, WindowsSize=0. 5. meta.AdaBoostM1, with options: classier=J48, Debug=false, NumIterations=10, useResampling=false, WeightThreshold=100. 6. meta.Bagging, with options: classier=J48, Debug=false, NumIterations=10, bagSizePercent=100, calcOutOfBag=false, Seed=1. 7. trees.J48, with options: binarySplits=false, CondenceFactor=0.25, debug=false, MinNumObj=2, numFolds=3, reducedErrorPruning=false, SavaInstanceData=false, Seed=1, subtreeRaising=true, Umpruned=false, UseLaplaze=false. The results of the experiments are shown in Table 2. The underline values indicate a signicant worse accuracy compares with the multi layer perceptron that has been used as the base of the comparison. We can see that all the classiers have an accuracy higher than 90% for all the data sets, what shows the benets of all the preprocess and normalization work. Note, thought, that the best results are obtained when the character is used without any characteristic extraction (a). The worse set of characteristics are the horizontal and vertical projections (b). Regarding the classiers, as we could expect, AdaBoost and Bagging improve the performance of the J48 that they are using as base learner, being AdaBoost better than Bagging. The best classiers is the SMO using directly the binary matrix that represent the character. However, nally we decide to use the multi layer perceptron, it is only a bit slower but is faster and its memory requirements are more than three times lower.

CONCLUSION

kernel methods: support vector learning, pages 185 208. MIT Press. Rodrguez, F. M. (1999). Contribuci n al reconocimiento o de caracteres en im genes complejas. PhD thesis, Esa cuela T cnica Superior de Ingenieros de Telecomuicae ciones, Campus Universitario, 36310, Vigo, Galicia, Spain. Serra, J. (1982). Image analysis and mathematical morphology. Academic Press, London, 2nd edition. Setchell, C. J. (1997). Applications of Computer Vision to Road-trafc Monitoring. PhD thesis, Faculty of Engineering, Department of Electrical and Electronic Engineering. Sirithinaphong, T. and Chamnongthai, K. (1999). The recognition of car license plate for automatic parking system. In Fifth International Symposium on Signal Processing and Its Applications, ISSPA, volume 1, pages 455457. Urios-Mond jar, D. (2007). e Sitio de las matrculas espa olas. http://www.geocities.com/amoros29/. (last n visited: 2-Dec-2007). Witten, I. H. and Frank, E. (2002). Data Mining: PRactical Machine Learning Tools and Techniques. Morgan Kauffman, 500 Sansome Street, Suite 400, San Francisco, CA 94111, 2nd edition. Zheng, L. and He, X. (2006). Number plate recognition based on support vector machines. In IEEE International Conference on Video and Signal Based Surveillance (AVSS06), page 13, Washington, DC, USA. IEEE Computer Society.

We have described an articial vision system used to recognize the Spanish cars license plate numbers in raster images. We combine the use classic image processing techniques with some new ideas, such as the soften lter and the character segmentation method. In the study of classiers we conrmed the benets of the preprocessing stages achieving accuracies above 90% for different sets of characteristics. For two of the classiers we increase the accuracy to 99%. In future version, we expect to take into account the two row Spanish license plates, and the special license plates with different combination of background and foreground colors. As well, as one of the reviewers suggested, we want to prepare a more detailed study of the use of the soften lter, specially regarding its robustness to noise in comparison with other lter in the literature.

ACKNOWLEDGEMENTS
This work has been possible thanks to the project BU004B06 from the Consejera de Educaci n de la o Junta de Castilla y Le n, Spain. o

REFERENCES
Chengwen, H., Yannan, Z., Jiaxin, W., and Zehong, Y. (2006). An improved method for the character recognition based on svm. In 24th IASTED International Conference on Articial Intelligence and Applications, pages 457461, Anaheim, CA, USA. ACTA Press. Draghici, S. (1997). A neural network based articial vision system for license plate recognition. In International Journal of Neural Systems, volume 8, pages 113126. Emiris, D. E. and Koulouriotis, D. E. (2001). Automated optic recognition of alphanumeric content in car license plates in a semi-structured environment. In International Conference on Image Processing, ICIP, volume 3, pages 5053. Gonzalez and Woods (2002). Digital Image Processing. Addison-Wesley, Reading, MA, USA. Nijhuis, J., Brugge, M., Helmholt, K., Pluim, J., Spaanenburg, L., Venema, R., and Westenberg, M. (1995). Car license plate recognition with neural networks and fuzzy logic. In IEEE International Conference on Neural Networks, volume 5, pages 22322236. ACTA Press. Platt, J. (1999). Fast training of support vector machines using sequential minimal optimization. In Scholkopf, B., Burges, C., and Smola, A., editors, Advances in