VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)

ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

A Review of Handwritten Text Recognition using Machine

Learning and Deep Learning Techniques
Priyank Shah1, Nitiket Shinde2, Deep Limbad3, Ashwini Save4
1
(Department Computer Engineering, Mumbai University, MUMBAI)
2
(Department Computer Engineering, Mumbai University, MUMBAI)
3
(Department Computer Engineering, Mumbai University, MUMBAI)
4
(HOD, Department Computer Engineering, Mumbai University, MUMBAI)

Abstract : Handwriting recognition has achieved tremendous success in the real world. It has various
applications such as equation solver, automation etc. Researchers have proven how recognition of handwritten
language characters is done by using a three step procedure using Machine learning algorithms. Moreover OCR
method allows to convert data from physical to digital. Some researchers have also discovered that colossal
amounts of handwritten documents must be digitally available when this handwritten recognition manifests, as
machines must understand the handwritten text. With technical advancements, researchers found better efficiency
using the deep learning model than machine learning models. In this paper survey is done to show which
algorithms are useful for handwritten text recognition, as many machine learning algorithms have shown that
handwritten strings under complex conditions cannot be accurately recognized where deep learning algorithms
work perfectly with high precision big data.
Keywords - Handwritten Text Recognition, Machine Learning, Deep Learning, Optical Character Recognition
(OCR).

I. INTRODUCTION
In today's world of automation self-learning of Machines and computers is getting more important and
necessary. The term Machine learning is a growing technology that enables machines like computers to learn
automatically from the previous data available. Deep learning is a subset of machine learning which on the other
hand is a subset of artificial intelligence. artificial intelligence is the general term that refers to techniques that
enable computers to mimic human behavior.
Various researches have been done on machine learning and deep learning with handwritten text
recognition. Many authors have studied that using various algorithms in machine learning like K-Means, SVM
with PCA, etc. can be used to recognize handwritten text where SVM is used with Euclidean distance as a norm.
Although SVM and K-means are mostly used machine learning algorithms for handwritten text recognition, there
are researches available which use machine learning TensorFlow which uses a machine-learning TensorFlow-
based platform for recognition. With TensorFlow neural networks comes into the picture and most of the popular
algorithms in the neural network which is used for image processing is Convolutional Neural Networks(CNN)
along with RNN layer, CTC loss, and CTC record. With the advancement in technology, CNN is improved to its
higher versions known as R-CNN that is a region-based convolutional neural network and many more which are
also used to get high accuracy in handwritten text recognition.
Based on previous researches it has been found that machine learning algorithms for any kind of text and
handwritten text recognition or for digit recognition machine learning algorithms can give accuracy around 95 to
96% which is the highest among all machine learning algorithms but points to be noticed are when used neural
networks accuracy can go much higher up to 97% using algorithms like CNN and its higher versions.

II. REVIEW OF LITERATURE SURVEY

1
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

R. Kanniga Devi, et . al. [1] In the paper, a comparative study is done for the classification of handwritten
digit recognition using various machine learning algorithms. Multiclass Decision Forest, Multiclass Decision
Jungle, Multiclass Neural Network, Multiclass Logistic Regression. For performing experiments on various
machine learning algorithms, Microsoft Azure Machine Learning Studio software. The data set used is MNIST
containing the gray-scale image dataset containing digits from zero to nine. All images are size of 28*28 pixels,
a total of 784 pixels. The dataset has 785 columns, label of digit drawn by user and 4,200 rows of which 60% is
used in training dataset and 40% is used for testing.The results of all the machine learning algorithms in this paper
had overall accuracy of Multiclass Decision Forest 93%, Multiclass Decision Jungle 87%, Multiclass Neural
Network 97%, Multiclass Logistic Regression 94%. Thus, this paper has demonstrated the accuracy of machine
learning algorithms and higher performances has been given by the Multiclass neural network.
Akanksha Gaur, et . al. [2] have proposed a system whose next step is feature extraction in which region
based k-means clustering is used and the feature vector is created and used in the classification phase as input.
Third step is the classification process, for which support vector machines are used. Results using SVM are better
than results using Euclidean distance. Characters with a better performance than 75% performance percentage
using the Euclidean distance method. Computation overhead in the Support Vector Machine is less than the
Euclidean distance approach. So classification using SVM is better than classification using the Euclidean distance
approach. This paper presents handwritten Hindi characters recognition based on K-means clustering and SVM.K-
means clustering reduces the size of the feature vector so that computation becomes easy. Here results are
calculated using two approaches for classification, one is Euclidean distance and other is Support vector machine.
Results using SVM are better than results using Euclidean distance. Maximum achieved result using Euclidean
distance is 81.7%. SVM is used with a linear kernel and gives 95.86% results. This paper proves that even machine
learning algorithms like SVM with PCA can give better results with good accuracy as compared to deep learning
algorithms.
Anupama Sahu, et . al. [3] have proposed a system in which two machine learning algorithms Naive
Bayes and Decision Table are used for handwritten character recognition of Odia language by using a tool called
WEKA (Open source application) to collect the data which will be used for developing machine learning
algorithms. WEKA has a 4-step process to integrate different classification techniques into a data set for achieving
the better result. The process starts by first taking the images as an input from the data set and pre-processing it
can apply any one of the classification algorithms to the data set and observe the accuracy and execution time of
a given task. In this paper, 240 Noisy images with a fixed size of 60*60 of them are collected and stored in the
folder. 240 image samples containing 12 numbers of odia vowels with noise were collected from 20 students. The
noise is eliminated and labelled during the pre-processing step, after which machine learning algorithms Naive
Bayes and Decision Table are applied in the classification step. The result from Naive Bayes Algorithm was able
to classify 116 noisy instances clearly and 124 noise instances incorrectly. The result from the Decision Table
was able to classify 66 noise instances correctly and 174 noisy instances incorrectly. The accuracy achieved by
Naive Bayes is 81% and with noise 49% and the accuracy achieved by Decision Table is 86% with noise is
24%.This paper provides efficiency of various machine learning algorithms for recognition of the odia
handwriting.
Nitin Gupta, et . al. [5] have proposed in this system mainly two types of approaches used in handwritten
text recognition are artificial neural networks that are a and an and hidden Markov model.Almost all the institutes
and governments having large amounts of handwritten papers are created every day. This insistent use of
computers to interpret handwritten texts, also creates something that is searchable and editable. Handwritten text
recognition by tensor flow requires simple execution. It is resourceful to break up complex problems and reduce
the extent of human action by changing handwritten text documents into digital form. The main application areas
of these currently are FASTag Implementation in India and book translation from one language to another
digitally.The system using this model is the algorithm of machine learning that attains relatively high recognition
rates compared to previous systems for both the evolution of text lines in the points and that of the collection of
horizontal, vertical, and skew lines. Accuracy can be improved by reducing the different forms of handwriting
like normalizing contrast and normalizing skew, normalizing slant and size of the text or character in the
handwriting. This paper gives an idea about HTR using Tensorflow where the system has attained relatively high
recognition rates compared to previous systems for both evaluation of text line elements and that of collection of
horizontal, vertical and skewed lines.
Peiyu Ma, [6] has developed a Convolutional neural network composed of multiple layers of artificial
neurons. The primary layer sometimes extracts basic options reminiscent of horizontal or diagonal edges. This
output is passed on to the next layer that detects a lot of complicated features such as corners or combinatory
edges. In the proposed system a data set of 60000 samples out of which 55000 is used for training and 5000 will
be used for testing. The MLP is a network of seven layers and the input of picture is of size 32 * 32. Convolution
consists of three dimensional length, width and depth. Length and width are always set to some common values

2
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

which is the size of the convolution kernel generally 3 * 3 or 5 *5. The ReLU function has a highest accuracy of
99.8% with fast convergence. This function helps in shortening the trading time and speed up the process of
convergence. Thus in this approach by using CNN an activation function is used for finding out the best accuracy
that can be achieved by using CNN.
Junqing Yang, et . al. [7] in this paper have developed a novel handwriting text recognition algorithm
based on deep learning to improve the problems. In this paper, the method based on an object detection algorithm
(Faster R-CNN) finds a new dimension to study the problem. The algorithm sets two steps: First, preprocessing
the handwriting character based on Faster RCNN, second, character recognition based on the Convolutional
Neural Networks. While maintaining the accuracy, the detection speed is greatly improved, so it is an effective
method for object detection.In preliminary processing, the whole text is divided into word region and background
region, then finishing character segmentation based on Faster R-CNN, saving the word region image and sending
it to the next step. Similarly, in meticulous processing, the word region image is divided into character region and
background region based on Faster R-CNN, recognizing the letters based on CNNs. Finally, combining the results
of character recognition with their corresponding spatial coordinates, storing it for arranging letters to word, then
it is integrated into the text recognition result. Compared with traditional OCR, this paper shows good performance
on complex handwriting text recognition.
Chen Jun, et . al. [8] have provided a Faster RCNN (fast regional convolution) algorithm to automatically
classify and identify text, icon and other objects in complex documents.It enables fast and accurate separation of
seal area, text area and page number area from complex documentsThe Faster R-CNN text detection system is
composed of two parts - A Region Proposal Network part and Fast RCNN part.Therefore, it is necessary to
manually classify the target categories in the complex document image into three categories, namely, the stamp ,
text and page numbers to get a tag image that matches complex document detection.The convolution layer of
CNN uses the convolution window to slide on the image and obtain a series of feature graphs.In this project, the
training network in the Faster Rcnn framework uses the Resnet network.The structure of ResNet can accelerate
the training of ultra-deep neural networks very quickly, and the accuracy of the target detection is also greatly
improved.Resnet has a deeper neural network, and also has a faster training speed, which can achieve better
detection results.The number of Anchor boxes can be reduced from 6000 to 2000, which can greatly improve the
calculation speed.Based on the feature graph generated by CNN network, three rectangular Windows with
different sizes and three aspect ratios (1:1, 1:2,2:1) are generated, which is called Anchor box, The function of the
RPN layer is to extract the target region in the document image, mainly for producing region proposals. This paper
gives an idea about how faster RCNN works better than R-CNN using the Resnet network as it is improving target
detection.
Zuo Huahong, et . al. [9] has proposed a new type method of adhesive handwritten digit recognition
based on improved Faster RCNN that aiming at the low recognition accuracy of the traditional machine learning
algorithm which is susceptible to digital writing quality, inter-digital adhesion, random noise background and
other factors in the process of adhesion handwritten digit recognition, an new method based on improved fast
regional convolutional neural network(Faster RCNN) of adhesion handwritten digit recognition is proposed. The
commonly used algorithm of continuous digit string mainly extract hand-made features, and use the traditional
learning algorithm to train effective classifiers for detection and recognition. Most of these methods are limited
to specific data sets, and each component is optimized separately, hence the handwritten strings under complex
conditions can’t be recognized accurately. In faster RCNN, Proposed a new layer called ROI Pooling that extracts
equal-length feature vectors from all proposals (i.e. ROIs) in the same image. Compared to R-CNN, which has
multiple stages (region proposal generation, feature extraction, and classification using SVM), Faster R-CNN
builds a network that has only a single stage. In this paper, a new method of handwritten digit recognition and
detection using deep learning technology is proposed.
Rohan Vaidya, et . al. [11] has proposed a system where the English handwritten text is recognized by
Deep learning Convolutional Neural Network (CNN). The frontend app is built on android to take the image as
the input. The backend of the system is a neural network model which is trained using a tool such as TensorFlow
with OpenCV library. For training the model NIST database is taken which contains thousands of handwritten
characters. After image processing steps are carried out, segmentation is converting all the words in the line into
the individual character which will be given to the pre-trained neural network model. After which, the final
predicted text is sent back as a response to the user input image. The accuracy achieved by the model in this
system is 94%. This paper provides a similar approach to recognition of handwritten text using deep learning
CNN model with the app to take the images.
R. Reeve Ingle et al. [12] have described three main techniques to design a scalable handwritten text
recognition system. In the first one they have described the data generation pipeline and presented a series of
techniques to generate better data, Also In this paper they have made use of GRCL block for large data. Authors
have integrated the handwritten recognition model into the full text recognition system with the help of

3
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

augmenting the script identification model with an additional classification between printed text and handwritten
text. The dual-head model yielded the best accuracy on both printed and handwritten text compared to printed and
handwritten-specific models.
Dr. Bhushan Vidhale et al. [13] have designed a translator using MATLAB to beat the barrier of various
languages. In this research work, a purely handwritten digit recognition using machine learning model as well as
character recognition matlab model is used. Input is taken in English, Marathi and Gujarati text manually to the
interface or image of written text or handwritten text and output can be translated in English Language by
facilitating use of Optical Character Recognition (OCR) technique. So in this paper keeping MATLAB in the
center authors have got the best accuracy in multi-linguistic Handwritten text recognition with the help of OCR
which is around 99%.
Ntirogiannis et al. [14] has studied that document image binarization is of great importance in the
document image analysis and recognition pipeline since it affects further stages of the recognition process. The
evaluation of the binarization method aids in studying its algorithmic behaviour as well as verifying its
effectiveness, by providing qualitative and quantitative indication of its performance. They proposed a pixel-based
binarization evaluation methodology for historical handwritten aur machine printed document images.

III. ANALYSIS

Analysis table summarizes the research papers on the Handwritten text recognition. Various Machine
Learning , Deep Learning Algorithms along with other technologies like OCR are used for recognizing the text.
Below is a detailed description of various algorithms used in research papers.

Table 1: Analysis Table

Sr. No Title Technology Dataset Accuracy

1 Handwritten Hindi Character K-means clustering Hindi Handwritten 95.86%

Recognition using K-Means and SVM characters
Clustering and SVM[2].

2 Odia Handwritten Character Machine Learning Odia Handwritten 1. 81%

Recognition with Noise 1. Naive Bayes Characters 2. 86%
using Machine Learning[3]. 2. Decision Trees

3 Handwritten Mathematical SVM CHROME Dataset 92%

Symbol Recognition Using
Machine Learning
Techniques: Review[4].

4 Machine Learning Tensor Tensorflow IAM Dataset. 95.29%

Flow Based Platform for
Recognition of HandWritten
Text[5].

5 Recognition of Handwritten Convolutional NIST Dataset. 92%

Digit Using Convolutional Neural Network
Neural Network[6].

6 Handwriting Text Faster R-CNN. EMNIST, Letter 97%

Recognition Based on Faster Dataset, Word
R-CNN[7]. Dataset

7 Automatic classification and Faster R-CNN. Wei Bo company 89.40%

4
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

recognition of complex electronic

documents based on Faster document.
RCNN[8].

8 A New Type Method of Faster R-CNN. CIFAR-100, 87.50%

Adhesive Handwritten Digit SVHN, Imagenet
Recognition Based on
Improved Faster RCNN[9].

9 Multilingual Text & CNN MNIST 99%

Handwritten Digit
Recognition and Conversion
of Regional languages into
Universal Language Using
Neural Networks[13].

10 A Performance Evaluation Document Image Random 91.50%

Methodology for Historical Binarization, OCR Documents
Document Image
Binarization[14].

11 Scene Text Detection and CNN Unnamed 81.0

Recognition: The Deep (Different
Learning Era[15]. Images)

Selective papers have been studied for this survey and after analyzing these papers they brought us up to
the conclusion that whenever Machine learning algorithms are applied for Handwritten text, digits recognition
they are giving a very good accuracy maximum around 92% if considered SVM and when Deep Learning used
for the same we can increase a accuracy much upto 99% which is a great advancement and algorithm or model is
giving such high accuracy is Convolutional Neural Networks (CNN) and its higher versions like Region based
Convolutional Neural Networks(R-CNN).

IV. CONCLUSION

With increase in digitalization of handwritten text around the world, it becomes increasingly important
to have the best algorithm for detecting handwritten text accuracy with a very low error rate. Various Machine
Learning algorithms and Deep Learning algorithms have been used for recognition of handwritten text. Among
all different algorithms, CNN has been proved to be more accurate for the classification of handwritten text. After
analyzing various papers, it can be concluded that CNN is able to yield better accuracy and result when compared
with others. But CNN also has some limitations if the CNN has several layers then the training process takes a lot
of time to compute if GPU is not available on the computer, Overfitting, class imbalance are the major challenges
while training the model. CNN is applied to a lot of applications such as Biometric Authentication, Image
classification, Medical Image Computing for Predictive Analytics, Healthcare Data Science, Facial recognition.

REFERENCES
Proceedings Papers:
[1] R. Kanniga Devi; G. Elizabeth Rani, “A Comparative Study on Handwritten Digit Recognizer using Machine Learning Technique”,
2019 IEEE International Conference on Clean Energy and Energy Efficient Electronics Circuit for Sustainable Development
(INCCES) , 18-20 Dec. 2019, Krishnankoil, India .

[2] Akanksha Gaur, Sunta Yadav,”Handwritten Hindi Character Recognition using KMeans Clustering and SVM”, IEEE 2015 4th
International Symposium on Emerging Trends and Technologies in Libraries and Information Services, pp. 1-6.

5
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

[3] Anupama Sahu,S. N. Mishra, “Odia Handwritten Character Recognition with Noise using Machine Learning” IEEE 2020
International Symposium on Sustainable Energy, Signal Processing and Cyber Security (iSSSC), Dec 16-17, 2020,pp.1-4.

[4] Syeda Aliya Firdaus, K. Vaidehi, “ Handwritten Mathematical Symbol Recognition Using Machine Learning Techniques:
Review”, Stanley College of Engineering and Technology for Women, Hyderabad,2020..

[5] Nitin Gupta, Neha Goyal,”Machine Learning Tensor Flow Based Platform for Recognition of HandWritten Text”, IEEE 2021
International Conference on Computer Communication and Informatics (ICCCI -2021), Jan. 27-29, 2021, Coimbatore,
INDIA,pp.1-6.

[6] Peiyu Ma,“Recognition of Handwritten Digit Using Convolutional Neural Network”,IEEE 2020 International Conference on
Computing and Data Science (CDS),2020,pp. 1-8.

[7] Junqing Yang, Peng Ren, Xiaoxiao Kong,“Handwriting Text Recognition Based on Faster R-CNN”, IEEE 2019 Chinese
Automation Congress (CAC), 2019, pp. 1-5.

[8] Chen Jun, Yang Suhua, Jiang Shaofeng,”Automatic classification and recognition of complex documents based on Faster RCNN
“IEEE 2019 14th International Conference on Electronic Measurement & Instruments.

[9] Zuo Huahong,Tang Junyi “A New Type Method of Adhesive Handwritten Digit Recognition Based on Improved Faster RCNN”,
IEEE 2020. 5th International Conference on Signal and Image Processing (ICSIP).

[10] Shifat Nayme Shuvo,Fuad Hasan”Handwritten Polynomial Equation Recognition and Simplification Using Convolutional Neural
Network ”, IEEE 2020,11th International Conference on Computing, Communication and Networking Technologies(ICCCNT).

[11] Rohan Vaidya, Darshan Trivedi, Sagar Satra, Prof. Mrunalini Pimpale, “Handwritten Character Recognition Using
Deep-Learning”, IEEE 2018, 2nd International Conference on Inventive Communication and Computational Technologies
(ICICCT), 20-21 April 2018, pp. 772-775.

[12] R. Reeve Ingle, Yasuhisa Fujii, Thomas Deselaers, Jonathan Baccash, Ashok C. Popat . “A scalable handwritten text recognition
system”.2019 International Conference on Document Analysis and Recognition (ICDAR) CoRR, abs/1904.09150, 2019.

[13] Dr. Bhushan Vidhale, Ganesh Khekare, Chetan Dhule, Dr. Pankaj Chandankhede, Abhijit Titarmare, Meenal Tayade
“Multilingual Text & Handwritten Digit Recognition and Conversion of Regional languages into Universal Language Using
Neural Networks”, 2021 6th International Conference for Convergence in Technology (I2CT) Pune, India. Apr 02-04, 2021.

[14] Ntirogiannis, Konstantinos, Basilis Gatos, and Ioannis Pratikakis. "A Performance Evaluation Methodology for Historical
Document Image Binarization.," IEEE International Conference on Document Analysis and Recognition, 2013.

Journal Papers:

[15] Shangbang Long, Cong Yao. “Scene Text Detection and Recognition: The Deep Learning Era ”, International Journal of
Computer Vision, Issue 1/2021, 14th April 2020.

6
www.viva-technology.org/New/IJRI