Synopsis on

HANDWRITTEN DIGIT RECOGNITION

USING CNN

Submitted by

Himanshi Jain

For the award of the degree of

Master of
Computer Application

Under the supervision of

Dr. Neelam Sharma

Associate Professor

Apaji Institute of Mathematics and Applied

Computer Technology
Banasthali Vidyapith
Banasthali – 304022
Session: 2019-20
 Table of Contents

S.No. Title Page No.

1. Introduction 1-2
1.1Organization
1.2 Problem Definition

2. Literature Review 3-8

3. Proposed Study 9
3.1 Aims and Objectives

4 Research Methodology

4.1 Methodology 10-12

4.2 Work Plan 13
4.3Proposed Contents of Thesis 14
4.4 Tools and Techniques 15

5 References 16
1. INTRODUCTION

1.1 ORGANISATION INTRODUCTION

Banasthali Vidyapith is a fully residential women’s deemed university. It is a higher
education institute which offers the integrated system extending from primary to Ph.d.
level.

Banasthali was founded by Smt.Ratan Shastri and Pandit Hiralal Shastri on October 6,
1935 to fill up the vacuum caused by the death of their highly talented and promising
daughter Shantabai. They had high expectations from Shantabai that she would work for
women’s when she grow up. But destiny ordained otherwise.

Banasthali is one of the five higher education in India meant exclusively for women.
Banasthali has developed into a National center for women’s education over these 75
years. Banasthali’s educational programme aims at an all-round development of student’s
personality. To achieve its ‘synthesis of spiritual values and scientific achievements of
the east and the west’ it has evolved Five-Fold educational programme Panchmukhi
Shiksha comprising of the following aspects: 1)Physical, 2)Practical, 3)Aesthetic,
4)Moral and 5)Intellectual. In this way the students developed an integrated and balanced
personality.

The Rural ambience of Banasthali Vidyapith is very conducive for promoting the
ideology of simple living and for imbibing and internalizing the values of self-
reliance and tolerance.

Following the philosophy of balanced Panchmukhi system of education, the institude

provides ample opportunities for integrated development of personality. The stress on the
positive aspects of the Indian heritage has resulted in a climate which has contributed
significantly to the task of upliftment of women to enable them to take on leadership
roles with confidence. Banasthali has been accredited by National Assessment and
Accreditation Council (NAAC) with ‘A’ Grade (Five Start rating). Banasthali is the
recipient of SANTBAL Award in the year 2000 for the services in the field of all round
education of women.

1.2 PROBLEM DEFINITION

Handwritten Digit Recognition plays a vital role in the world of machine learning and
deep learning. Since 1980’s it is an important topic in machine learning. Each and every
individual has a different style of writing and it found difficult in recognizing handwritten
digits by humans. Each handwritten digit written by different people has different
direction, shape, style, etc. and it has become a challenging task for researchers. Many
machine learning and deep learning models were proposed for recognition of handwritten
digits. For recognition of handwritten digits machines are included because machines can
identify digits using different methods. First these handwritten characters are converted to
digital format because machines can only understand binary characters. MNIST dataset is
used for recognition of digits which contains 70,000 digit images (0-9), size of the images
are 28*28 pixels. A system had been designed using machine learning and deep learning
algorithms for classification. But in advanced researches deep learning algorithms
perform better than machine learning algorithms. Results with deep learning are more
accurate with less error rate. But with some pair of digits machine also finds difficulty in
recognition of digits correctly.

Handwritten digits are not always of similar size, shape, width and justified to margins,
these digits will always vary from person to person, so the general problem while
classifying the digits would be similarity between digits like 1 and 7, 3 and 8, 2 and 5, 2
and 7, etc. When many people write a single digit with variety of handwritings this type
of problem is faced, even machines get confused with different type of handwritten
digits. The formation of digits is influenced by the uniqueness and variety in the writing
of every individual.

Nowadays, digital document is trying to replace paper work, but still we see paper work
in most of the places in our daily life. Machines cannot convert the handwritten digit
written on physical paper directly. First these handwritten characters are converted to
digital characters and it is a tough task which has been continuing from past. Many
researchers have worked on this topic and multiple techniques have been proposed for
handwritten digit recognition.
2. LITERATURE REVIEW

2.1 RELATED WORK

Using machine learning and deep learning algorithms several implementation has been
done by many researchers on handwritten digits images. Handwritten digit recognition is
the vast area for research in the field of classification and recognition. Now, we will
discuss some related works of researchers who has proposed different frameworks for
classification of handwritten digits.

Siddhartha Roy et. al (2017) in his paper explains the importance of choosing right
approach for feature extraction and classification for Optical Character recognition or
Hand-Written Character recognition [1]. He has used 10 features for recognition of
handwritten numerals and for classification he has proposed a model with KNN (K
Nearest Neighbor) classifier and gains the accuracy of 97.67% in the process of
recognition. This work is proposed only for Handwritten numerals.

Meer Zohra et. al (2019) has shown the comparison between SVM (Support Vector
Machine), KNN (K Nearest Neighbor) and CNN (Convolution Neural Network)
classifier in the paper [2].CNN has achieved 99.4% accuracy on trained dataset and
98.4% accuracy on test dataset.KNN has achieved the accuracy of 97.1% on train dataset
and 96.7% on test dataset.97.9% trained data accuracy and 97.6% test data accuracy has
obtained for SVM classifier. It concludes that CNN performs better than the other two
classifiers. Research work is proposed for handwritten numerals.

Xiao-Xiao Niu et. al (2012) has presented a novel hybrid CNN-SVM classifier for
recognition of handwritten digits [3]. On comparison to other studies this fusion has
obtained a better recognition rate.

P.D. Gader et. al (1996) had explained an automatic feature generation method for
handwritten digit recognition and these features are then used in backpropagation trained
neural network [4]. This work is performed on handwritten digit images.
Kh Tohidul Islam et. al (2017) in his work used artificial neural network for classification
and image pixels for feature extraction [5]. He has compared his results with SVM+HOG
(feature extraction method), R-HOG+SVM, HOG+ANN, etc. But he achieved his best
results with image pixels + ANN. This work is implemented with handwritten digit
images.

Saeed AL-Mansoori et. al (2015) his paper presented a multilayer perceptron Neural
Network for Handwritten digit problems [6]. The performance of the system was
observed with various number of hidden units and number of iterations. Maximum
training accuracy of 99.32% and testing accuracy of 100% was obtained with 25 hidden
neurons and 250 iterations.

Shengfeng Chen et. al (2018) had worked on 5 machine learning algorithms Neural
Network, K-Nearest Neighbor, Decision Tree, Random Forest and Bagging with gradient
boost [13]. The experimental results shows that KNN has similar higher accuracy of
96.7% on comparing with Neural Network of 96.8%. However KNN has 10 times faster
processing speed. According to analysis of this paper KNN combined with preprocessing
methods achieved a great performance apart from Neural Network when used as a
classification algorithm in offline handwritten digit recognition.

2.2 CONVOLUTION NEURAL NETWORK

Convolution neural network is a highly used method specially for image processing as it
has inbuilt feature extraction technique, and it gives more accurate results than other
machine learning algorithms and simple neural network. We will now discuss some of
the works related to CNN.

Vijayalaxmi R Rudraswaminath et. al (2019) has presented a research with multi layer
CNN and other machine learning algorithms like KNN, SVM and RFC [7]. She has
concluded that on comparison to KNN the results obtained from CNN are better and
results obtained with SVM and RFC are very much lower than KNN. And the presented
model is using Keras+Theano as backend and Tensorflow libraries. This research work is
presented for handwritten digit images.

Fathma Siddique et. al (2019) had presented a research on handwritten digit recognition
utilizing CNN with Tensorflow and compares the results for various hidden layers [8].
Maximum accuracy is obtained with the combination of Conv1, pool1, Conv2, pool2
hidden layers with two dropout layers.

Andr´es Calder´on et. al (2003) in his work presented a multilayer feedforward network
known Convolution neural network, a novel convolution neural network using gabor
filters at the first layer for feature extraction is implemented in this model [9]. He has
used a backpropagation method for training. All this work was done on handwritten digits
images.

Ram Sarkar et. al (2019) had worked on the reduction of feature maps so that they
consume less storage space and they consume less computation time [10]. This technique
reduces the computation time for CNN without affecting the accuracy.

T Siva Ajay et. al (2017) had worked on LeNet architecture of CNN [11]. Factors
responsible for getting higher accuracy such as low memory requirements, low run time
and better accuracy, this paper primarily focuses on getting better accuracy rate for
classification. LeNet architecture is used for better accuracy results, aim of this paper is
to deal with one of the method in which it can be implemented, several methods are there
in which this implementation can be done using different frameworks like octave, matlab.

Haider A. Alwazwazy et. al (2016) had worked on the Arabic dataset which was trained
using CNN model as it is the most efficient model for variety of applications [12]. They
have used the best open source of deep learning called Caffe for their experiment. High
performance Graphical Processing Units (GPUs) which had 1200 cores is utilized for
experiment, so that the training and testing are fast satisfying the real time applications.
This network was trained with (10,000) iteration and 128 samples of images is used in
each batch. Gaussian distribution for random initialization of CNN parameters and
learning rate parameter are smoothly decreasing after each epoch.
3. PROPOSED STUDY

3.1 AIMS and OBJECTIVES

Aim of this project is to achieve the better accuracy with CNN (Convolution Neural
Network) utilizing Keras and Tensorflow in backend and those results are than compared
with some machine learning algorithms like KNN (K Nearest Neighbor), SVM (Support
Vector Machine), RFC (Random Forest Classifier) and deep learning method ANN
(Artificial Neural Network) using application Handwritten digit recognition

The main objectives and goals of this project are:

1. Data preprocessing has been done for normalization and reshaping.

2. Relevant features will be extracted using feature extraction techniques.

3. Machine learning and deep learning techniques will be applied for training and
classification of the model.

4. Evaluation of the model has to be done with testing dataset.

5. And in last step we will compare the algorithms according to accuracies.

4. RESEARCH METHODOLOGY

4.1 METHODOLOGY

4.1.1 Flow Chart

Figure 4.1: Flow chart for steps performed for Recognition

4.1.2 Data Preprocessing

The MNIST dataset is already preprocessed with its background and foreground inverted,
images are in the form of binary code, and the digits outline of white color, edge
thickening, removal of noise and they all are already resized in a square shape. These
images are already in gray scale form. For pattern recognition/machine learning tasks this
dataset has been used, for that the background is converted to black and the characters are
converted to white. However, for more accurate results we use normalization method in
which we divide all images by 255. The main purpose of preprocessing is to smooth and
normalize the input data.
4.1.3 Feature Extraction

Feature extraction forecast a set of features from high dimensional space into a reduced
set of features from a low dimensional space in machine learning while explaining the
data with sufficient accuracy. For extracting relevant information from image data several
feature extraction techniques will be used which helps in recognition of digits more
accurately. In this work a total of three feature extraction techniques PCA, LDA and ICA
will be used. After data preprocessing feature extraction will be performed for extracting
the relevant features.

4.1.4 Classification techniques

Machine learning techniques such as KNN, SVM and RFC are used for classification of
digits. For machine learning classification data preprocessing + feature extraction + KNN
classifier model is utilized for recognition of digits, and the same method is applied with
SVM and RFC and in deep learning preprocessing + CNN model is applied for
classification, similar method is used for ANN.

KNN (K-Nearest Neighbor): This algorithm apply on classification and regression

problems, as it is simple and also it is easy-to-implement supervised learning algorithm,
and it is also known as a instance based learning algorithm. It is a lazy and also a non
parametric learning algorithm as it uses full data and imagines nothing about the present
data. For gaining excellent performance the algorithm requires a set of trained dataset in
which data points are accurately labeled. This algorithm takes new data points as input and
calculates the distance between new and labeled data points utilizing Euclidean distance
for classification. K-NN uses K for number of votes which is used for decision making
and for eliminating the connection between two groups odd value for K is selected.
Formula for Euclidean distance calculation is:

( ) ∑( )
SVM (Support Vector Machine): The support vector machine is a supervised learning
algorithm. And it can work for both the classification and the regression task, but it is
largely used for classification. Every input data item is represented as a point in space,
and they are mapped and classified into several classes. Here, we perform classification
and this algorithm separates the classes through the hyper-plane concept which
differentiates the classes very well and the margin should be equidistant from both the
classes. SVM is mostly used for the classification of two classes. However, it can also be
used for multi-class problems known as the one-against-rest approach. It is also
considered as a more advanced tool for solving linear and non linear classification
problems due to its simplicity, flexibility, prediction capability and optimality.

RFC (Random Forest Classifier): Classification and regression both duties can be
performed by RFC. It is a type of supervised learning method. To observe the more
accurate results for RFC, it needs large number of trees. This algorithm will avoid the over
fitting issues when there are enough number of trees and it never over fits model. It can
also work with missing values. The trees of the forest are created by picking out samples
randomly from the data set. The predictions of each tree are recorded and the best solution
is opted through voting. After training the model, results are observed for each individual
tree and then the average is calculated.

ANN (Artificial Neural Network): The architecture of ANN is same as that of neural
network.It consists of a irregular mapping architecture and it is the form of a collection of
nodes or artificial neurons. They are highly capable of predicting values, recognising
pattern,compressing data,decision-making,etc.Recently ANN is utilized for
classification.Different architectures for ANN are introduced.It consists of three layers
the input layer,hidden layer and output layer.The hidden layer and input layer are
connected via weights.

CNN (Convolution Neural Network): First CNN framework was propounded by

Fukushima in the year 1980. Hence, at that period of time it was not considered to be
used for training, since people find it difficult working with CNN.And then in 1990’s
Yann LeCunn proposed a new approach gradient-based learning algorithm to CNN and
observed successful results.And then after this approach priority is gained by CNN.
Almost everywhere in the field of pattern recognition CNN is utilized for recognising
patterns and much better results are recorded with CNN.As CNN is the multilayer feed
forward neural nework with deep supervised learning architecture,it is used for solving
complex jobs like text,sounds,images,vedio,etc. CNN is the combination of two parts
automatic feature extractor and trainable classifier. Feature extraction is performed
automatically with CNN.Much less preprocessing is needed in this algorithm than other
classification algorithms, it is capable of learning features. Since, it was inspired by the
connectivity pattern of neurons which has learnable weights in animal visual cortex.

CNN consists of three layers input layer,hidden layer and output layer.Convolutional
layer, ReLU layer i.e. activation function, pooling layers, fully connected layers, and
normalization layers, all are hidden layers.

4.1.5 Comparison

Comparison between algorithms is being performed according to their accuracies obtained

after recognition. It will show that which algorithm is performing more accurately.
4.2 WORK PLAN

First Month (1-4 Weeks): First 4 weeks are utilized in studying about the topic or
problem, literature survey and learning about the dataset as the dataset is freely available
on internet and creation of dataset is not required.

Second Month (5-8 Weeks): All four weeks of this month are used in learning and
applying data preprocessing techniques and feature extraction techniques, including tools
learning.

Third Month (9-12 Weeks): Training Model: Only three weeks of this month were
utilized in training model with training dataset and testing is performed with testing
dataset using machine learning techniques at initial level using different feature extraction
methods.

Fourth Month (13-16 Weeks): Training Model: Deep learning methods are also used so
for model training and testing with deep learning this month is used.

Fifth Month (16-19 Weeks): Three weeks of this month will be used in evaluating test
results with accuracy and error rate.
4.3 PROPOSED CONTENTS Of THESIS

The Thesis of this project includes the following chapters:

CHAPTER 1: Introduction: The main focus of this chapter is on describing the problem
statement of Handwritten digit recognition and the importance of the topic.

CHAPTER 2: Literature Review: In this chapter we will gather the information about
the related works done by various researchers and extract the useful information from
them.

CHAPTER 3: Research Objectives: This chapter explains about the objectives of the
research.

CHAPTER 4: Methodology: This chapter includes the data preprocessing, feature

extraction and classification approaches for handwritten characters.

CHAPTER 5: Results and Discussion: This chapter includes the experimental results
performed on different classification techniques and shows the comparison between the
results obtained from different classification techniques.

CHAPTER 6: Conclusion: In this chapter we conclude about the work results and the
problems addressed and limitation and it also suggests about the future work.
4.4 TOOLS and TECHNIQUES

This project will be developed by using python tool. The experiments performed for this
work are data preprocessing, feature extraction, training model and evaluating it with
testing dataset. Machine learning and deep learning techniques were used for
classification.

Data will be preprocessed and then the relevant features are extracted from the dataset.
Then finally this dataset is used for training the model and then it is evaluated using
testing dataset, but in deep learning algorithms we are not using feature extraction
techniques. Finally the results are evaluated on the basis of accuracy and loss rate.
5. REFERENCES

1. Siddhartha Roy and M.Saravanan (2007) ‘Handwritten Character Recognition using K-

NN Classification Algorithm’ (IJARIIE) Vol-3 Issue-5, ISSN (O)-2395-4396.

2. Meer Zohra and D.Rajeswara Rao (April 2019) ‘A Comprehensive Data Analysis on
Handwritten Digit Recognition using Machine Learning Approach’ International Journal
of Innovative Technology and Exploring Engineering(IJITEE), Volume-8 Issue-6, ISSN:
2278-3075.

3. Xiaoxiao Niu and Ching Y. Suen (April 2012) ‘A novel hybrid CNN–SVM
classifier for recognizing handwritten digits’ , Center for Pattern Recognition and
Machine Intelligence, Concordia University, Vol 45, Issue 4, pp 1318-1325.

4. P.D. Gader and M.A. Khabou (1996) ‘Automatic Feature generation for
Handwritten Digit Recognition’ IEEE Transactions on Pattern Analysis and
Machine Intelligence, Vol 18, Issue 12.

5. Kh Tohidul Islam, Ghulam Mujtaba, Dr. Ram Gopal Raj and Henry Friday (2017)
Nweke ‘Handwritten Digit Recognition using Artificial Neural Network’
International Conference on Engineering Technologies and Technopreneurship
(ICE2T).

6. Saeed AL-Mansoori (2015) ‘Intelligent Handwritten Digit Recognition using

Artificial Neural Network’ International Journal of Engineering Research and
Applications, Vol.5, Issue 5, pp.46-51.

7. Vijayalaxmi R Rudraswamimath, Bhavanishankar K (2019) ‘Handwritten Digit

Recognition using CNN’International Journal of Innovative Science and Research
Technology, Vol.4, Issue 6.
8. Fathma Siddique, Shadman Sakib and Md. Abu Bakr Siddique
(2019)‘Recognition of Handwritten Digit using Convolutional Neural Network in
Python with Tensorflow and Comparison of Performance for Various Hidden Layers’,
5th International Conference on Advances in Electrical Engineering (ICAEE)
2019:IEEE.

9. Andres Calderon, Sergio Roa and Jorge Victorino (2003) ‘Handwritten Digit
Recognition using Convolutional Neural Networks and Gabor Filters’
International Congress on Computational Intelligence (CIIC).

10. Sinjan Chakraborty, Sayantan Paul, Ram Sarkar and Mita Nasipuri (2019)
‘Feature Map Reduction in CNN for Handwritten Digit Recognition’ Kalita J.,
Balas V., Borah S., Pradhan R. (eds) Recent Developments in Machine
Learning and Data Analytics. Advances in Intelligent Systems and Computing,
Vol 740. Springer, Singapore.

11. T SIVA AJAY (2017) ‘Handwritten Digit Recognition using Convolutional

Neural Network’ International Research Journals of Engineering and
Technology (IRJET),Vol. 4, Issue 07, pp.2971-2976.

12. Haider A. Alwzwazy, Hayder M. Albehadili, Younes S. Alwan and Naz E.

Islam (2016) ‘Handwritten Digit Recognition Using Convolutional Neural
Network’ International Journal of Innovative Research in Computer and
Communication Engineering (IJIRCCE), Vol. 4, Issue 2, pp.1101-1106.

13. Shengfeng Chen, Rabia Almamlook, Yuwen Gu and Dr. Lee wells (2018)
‘Offline Handwritten Digits Recognition using Machine Learning’
International Conference on Industrial Engineering and Operations
Management, pp. 274-286.