2017 14th IAPR International Conference on Document Analysis and Recognition

Training an End-to-End System for Handwritten

Mathematical Expression Recognition by Generated
Patterns
Anh Duc Le and Masaki Nakagawa


Department of Computer Information

Tokyo University of Agriculture and Technology
Email: [email protected], [email protected]

Abstract—Motivated by recent successes in neural machine that presents all recognizable parses of the input. Then, the
translation and image caption generation, we present an end-to- extraction process finds the top to nth-most highly-ranked trees
end system to recognize Online Handwritten Mathematical from the forest. By using horizontal and vertical order, this
Expressions (OHMEs). Our system has three parts: a method reduces infeasible partitions and makes the method
convolution neural network for feature extraction, a bidirectional independent from stroke order. However, the worst-case
LSTM for encoding extracted features, and an LSTM and an number of sub-partitions that must be considered during
attention model for generating target LaTex. For recognizing parsing and the complexity of the parsing algorithm are still
complex structures, our system needs large data for training. We
quite large as O(n4) and O(n4|P|), respectively. This system
propose local and global distortion models for generating
OHMEs from the CROHME database. We evaluate the end-to-
incorporates a correction mechanism to help users to edit
end system on the CROHME database and the generated recognition errors.
databases. The experiential results show that the end-to-end A global approach allowing mathematical symbols and
system achieves 28.09% and 35.19% recognition rates on structural relations to be learned directly from expressions was
CROHME without and with the generated data, respectively. proposed by Awal et al. [5]. During the training phase, symbol
hypotheses are generated without using a language model. The
Keywords—Online Handwritten Mathematical Expression
dynamic programming algorithm finds the best segmentation
Recognition, End-to-End Model, Encoder-Decoder Model, Patterns
Generation
and recognition of the input. The classifier learns both the
correct and incorrect segmentations. The training process is
I. INTRODUCTION repeated to update the classifier until the classifier recognizes
the training set of OHMEs correctly. Furthermore, contextual
Recognition of online handwritten mathematical expression modeling based on structural analysis of the expression is
(OHME) is one of the current challenges concerning employed, where the models are learnt directly from
handwriting recognition. It can be divided into three main expressions using the global learning scheme.
processes. First, a sequence of input strokes is segmented into
hypothetical symbols (symbol segmentation). Then A formal model for OHME recognition based on 2D
hypothetical symbols are recognized by a symbol classifier Stochastic Context Free Grammar (SCFG) and Hidden Markov
(symbol recognition). Finally, structural relations among the Model (HMM) was proposed by Alvaro et al. [6]. HMM uses
recognized symbols are determined and the structure of the both online and offline features to recognize mathematical
expression is analyzed by a parsing algorithm in order to symbols. The Cocke-Younger-Kasami (CYK) algorithm is
provide the most likely interpretation of an input OHME modified to parse an input OHME in two dimensions (2D).
(structural analysis). The recognition problem requires not only They use the range search to improve time complexity from
segmentation and recognition of symbols but also analysis of O(n4|P|) to O(n3logn|P|). To determine structural relations
two-dimensional (2D) structures and interpretation of the among symbols and sub-expressions, a Support Vector
structural relations. Ambiguities arise in all stages of the Machine (SVM) learns geometric features between bounding
process. boxes.
Many approaches have been proposed for recognizing Le et al. presented a recognition method based on SCFG
OHMEs especially during last two decades. They are [7]. Stroke order is employed to reduce the search space and
summarized in the survey papers [1, 2] and the recent the CYK algorithm is employed to parse a sequence of input
competition papers [3]. Most of them follow three strokes. Therefore, the complexity of the parsing algorithm is
interdependent processes as mentioned above. These processes still O(n3|P|), like that of the original CYK algorithm. They
can be handled independently [2] or jointly [4, 5, 6, 7]. In the extended the grammar rules to cope with multiple symbol
following, we will review a few recent approaches participated variations and proposed a concept of body box with two SVM
in the recent Competition on Recognition of Online models for classifying structural relations. The experiments
Handwritten Mathematical Expressions (CROHME). showed the good recognition rate and practical processing
time.
A system for recognizing OHMEs by using a top-down
parsing algorithm was proposed by MacLean et al. [4]. The A modified version of the Minimum Span Tree (MST)
incremental parsing process constructs a shared parse forest based parsing algorithm was presented by Hu et al. [8]. The

2379-2140/17 $31.00 © 2017 IEEE 896

DOI 10.1109/ICDAR.2017.151
parser extracts MST from a directed Line-of-Sight graph. The this paper, the size of square is (8 x 8). CNN takes a square and
time complexity of this parsing method is lower than the time produces a feature vector with D elements. As a result, we
complexity of the CYK parsing method. This parser achieved obtain a sequence of feature vectors (F1, F2, ..., FKxL) from an
good result of structure analysis on OHME patterns assuming input image (H, W), where (H, W) and (K, L) are image sizes
correct segmentation and symbol recognition. and reduced sizes and D is the depth of features. The order of
the feature extraction is from left to right and from top to
Mouchere et al. have been organizing CROHME for fair bottom.
evaluation based on common databases. The above systems
have shown good performance for the recent CROHME B. Encoder
databases. However, they require ground-truth of OHMEs in An encoder encodes the sequence of feature vectors into a
different levels such as stroke, symbol, and structure. The sequence of outputs (E1, E2, ..., EKxL). We employ a
collection and preparation of ground-truth for OHMEs are bidirectional LSTM which contains a forward LSTM and a
time-consuming tasks. The CROHME training set currently backward LSTM.
contains 8835 OHMEs from five different databases. It is hard
to increase the number of OHMEs because it takes time and C. Decoder
effort to collect and make ground-truth. Recently, Zhang et al. A decoder generates one symbol at a time. At each time
proposed a method using BLSTM for interpreting 2D step t, the decoder predict symbol yt based on the current
languages such as OHMEs [9]. The method is an end-to-end output Ot, and the context vector Ct. Ot is calculated from the
model which requires only input data and their corresponding previous hidden state of the decoder htí1, the previous decoded
Latex or MathML. It is able to produce the result from input vector Ot-1, and the previous symbol yt-1. Ct is computed by
handwriting by using a BLSTM model. However, the weighted sum of the sequence of outputs and their weights
performance is still lower than the above systems. produced by an attention model.
Recently, an attention-based encoder-decoder model has
been successful in machine translation [10] and image caption
generation [11]. It outperforms traditional methods in many
tasks of sequence to sequence problems. Y. Deng et al.
extended this model to recognize images of printed MEs to
LaTex [12]. This model shows an encouraging result on
printed MEs patterns. For OHMEs, however, the problem is
hard since symbols and structures of OHMEs have more
variations and distortions than those of printed MEs.
In this paper, we present an end-to-end system employing
convolution neural network based on the attention-based
encoder-decoder model. In our knowledge, this is the first work
that employs the attention-based encoder-decoder for
handwriting recognition. This system requires a large data for
training, so we propose local and global distortion models to
generate OHMEs from the CROHME database.
The rest of this paper is organized as follows. The end-to-
end system for recognition of OHMEs is presented in Section
2. The local and global distortion models for data generation
are described in Section 3. The experimental results are
presented and discussed in Section 4. Conclusions are drawn in
Section 5.
II. OVERVIEW OF THE END-TO-END RECOGNITION SYSTEM
The structure of the end-to-end recognition system is
shown in Figure 1. It has three parts: a convolution neural
network for feature extraction from the image of an OHME, a
bidirectional LSTM for encoding extracted features, and a
LSTM and attention model for generating the target LaTex.
They are described in the following sections.
Fig. 1. Structure of the end-to-end model
A. Feature Extraction by CNN
Features are extracted from the image of an OHME by a III. PATTERNS GENERATION
convolution neural network which contains multiple layers of Patterns generation was successfully applied for Japanese
convolution and max-pooling layers. This is a standard CNN handwritten text recognition by Chen et al. [13]. In this work,
without recent techniques such as dropout, maxout, etc. An we extend their model into local and global distortions.
input image (H x W) is divided into (K x L) equal squares. In OHMEs are distorted by combination of local and global

897
distortions. The local distortion is applied for symbols in an šԢ ൌ š ‘•Ⱦ ൅ ›•‹Ⱦ
OHME, while the global distortion is applied for the whole ൜ (7)
› ᇱ ൌ š•‹Ⱦ ൅ › ‘•Ⱦ
OHME. The local distortion includes shear, shrink,
perspective, shrink plus rotation, and perspective plus rotation. where (x’, y’) is the new coordinate transformed by any of
The global distortion includes scaling and rotation. The local distortion models, Į is the angle of shear, shrink, and
process of distortion is shown in Figure 2. First, all symbols in perspective distortion models and ȕ is the angle of rotation
an OHME are distorted by the same distortion models. Then, distortion model. The local distortion model and its
the OHME is distorted by scaling and rotation models parameters are presented by (id, Į, ȕ) where id is the identifier
sequentially. The distortion models are described in the of the distortion model from 1 to 5, Į and ȕ are from -10o to
following sections. 10o.

Local distortion
Figure 3 show examples of local distortion models with Į
= 10o and ȕ = 10o.
Shear

Shrink Global distortion

Original Shrink +
Original Vertical shear Horizontal shear
ME Rotation Scaling Rotation
Distorted
Perspective ME

Perspective
+ Rotation

Fig. 2. Process of distortion model for patterns generation.

Vertical shrink Horizontal Vertical shrink Horizontal
A. Local Distortion shrink + rotation shrink +
rotation
Shear is a transformation that moves points in an axis by a
distance increasing linearly with the other axis. The shear
includes vertical and horizontal shear models. They are
calculated by Eqs. (1) and (2).
The shrink and perspective are both similar to the shear Vertical e Horizontal Vertical e Horizontal
with different transformation equations. The vertical and perspective perspective
perspective + perspective +
horizontal shrink models are described in Eqs. (3) and (4) rotation rotation
respectively. The vertical and horizontal perspective models Fig. 3. Examples of local deformation by shear, shrink and perspective
are shown in Eqs. (5) and (6), respectively. transformations.

The shrink plus rotation model applies shrink and rotation B. Global Distortion
models sequentially. It is similar to the perspective plus Global distortion distorts an OHME in baseline and size.
rotation models. The rotation model is shown in Eq. (7). We employ rotation and scaling models. The rotation model is
šԢ ൌ š ൅ › –ƒ Ƚ šԢ ൌ š similar to the local distortion. Scaling model is shown in Eq.
൜ (1) ൜ ᇱ (2) (8). An example of the global distortion is shown in Figure 4.
›ᇱ ൌ › › ൌ › ൅ š–ƒȽ
஠ ୶ ୱ୧୬ሺ஑ሻ šԢ ൌ š
š ᇱ ൌ ›ሺ•‹ ቀ െ Ƚቁ െ ቀ ቁሻ ൜ (8)
ቊ ଶ ଵ଴଴ ሺ3) › ᇱ ൌ ›
›ᇱ ൌ › where k is the scaling factor. The parameters of the global
šԢ ൌ š distortion model are presented by (k, Ȗ) where Ȗ is the angle of
ቊ ᇱ ஠ ୷ ୱ୧୬ሺ஑ሻ (4) the global rotation distortion model, k is from 0.7 to 1.3, and Ȗ
› ൌ šሺ•‹ ቀ െ Ƚቁ െ ቀ ቁሻ
ଶ ଵ଴଴ is from -10o to 10o.
ଶ ୶ିହ଴
šԢ ൌ ሺš ൅ ͷͲ ‘• ቀͶȽ ቁሻ
ଷ ଵ଴଴
ቐ ଶ గ ୷ ୱ୧୬ሺ஑ሻ
(5)
› ᇱ ൌ ›ሺ•‹ ቀ െ ߙቁ െ  ቀ ቁሻ
ଷ ଶ ଵ଴଴

ଶ గ ୶ ୱ୧୬ሺ஑ሻ
š ᇱ ൌ š ൬•‹ ቀ െ ߙቁ െ  ቀ ቁ൰
ଷ ଶ ଵ଴଴
ቐ ሺ6)
ଶ ୷ିହ଴ Scaling factor = 0.7 and rotation
› ᇱ ൌ ቀ› ൅ ͷͲ ‘• ቀͶȽ ቁቁ Original
ଷ ଵ଴଴ angle = 7o
Fig. 4. Examples of global distortion by scaling and rotation models.

898
C. Patterns generation Next, we compared the performance of the best end-to-end
To generate an OHME, we first randomize five variables system in the above with the other systems which participated
(id, Į, ȕ, k, Ȗ). Then, all symbols in an OHME are distorted by CROHME 2014.
local distortion models with (id, Į, ȕ). Then, the OHME is A. Databases
distorted by the global distortion model with (k, Ȗ). Figure 5
shows some generated OHMEs from the original OHME that We use the CROHME 2014 database [11]. Organized at
appeared in Figure 3. ICHFR 2014, CROHME 2014 was a contest in which OHME
recognition algorithms competed. It allows the performance of
the proposed system to be compared with others under the
same conditions. There were seven participants. The
CROHME 2014 database contains 8,835 OHMEs for training
and 986 OHMEs for testing. The number of symbol classes is
101.
We generated more patterns by using the above-mentioned
distortion models. We prepared two new training sets named as
G_CROHME1 and G_CROHME2. G_CROHME1 and
G_CROHME2 were created by generating 3 and 5 new
OHMEs from every OHME in the CROHME training set,
respectively. They also include original OHMEs from the
CROHME training set. The number of OHMEs and generated
OHMEs for each training set are shown in table I.

TABLE I. DESCRIPTION OF TRAINING SETS

CROHME G_CROHME G_CROHME
training set 1 2
Fig. 5. Sample OHMEs generated by combination of local and global # of OHMEs 8,835 35,340 53010
distortion models. # of generated
0 26,508 44,180
OHMEs

IV. EVALUATION
We employ the CROHME 2013 test set for validation and
First, we trained the end-to-end system by the CROHME the CROHME 2014 test set for evaluation.
training set. We repeatedly employ the training set to train the
system. The training terminates when no increase of Input
recognition rate is observed after 10 epochs. The resultant Conv (3x3): 64
system is referred as the baseline system. Then, we created the Batch Norm
two new training datasets G_CROHME1 and G_CROHME2 ReLU
by patterns generation which is detailed in the next section. For Max pooling (2x2)
each generated dataset, we trained the end-to-end system with
applying global distortions of different values for parameters at Conv (3x3): 128
Batch Norm
every epoch as shown in Figure 6(a). We also trained the ReLU
system without global distortion as shown Figure 6(b). Namely,
training with global distortions uses images from training set Max pooling (2x2)
with global distortions applied at the beginning of every epoch Conv (3x3): 256
while training without distortion employ the same images from Batch Norm
the training set for every epoch. Then, we evaluated all the ReLU
systems on the CROHME 2014 test set. Max pooling (1x2)

Conv (3x3): 256

Batch Norm
Training Training ReLU
Database Database
Max pooling (2x1)
Epoch k
Without Epoch k Conv (3x3): 512
Global
distortion distortion Batch Norm
ReLU

Training process Output

Training process

(a) with global distortions (b) without global distortions Fig. 7. Structure of CNN feature extraction. The parameters of the
convolution and max pooling layers are denoted as “Conv (filter size):
number of filters” and “Max pooling (filter size)”, respectively.
Fig. 6.The training process of the end-to-end model with and without
distortions.

899
B. End-to-end system configuration latex format, so that we obtain only expression recognition rate.
A CNN with convolution, batch norm, ReLU, and max- The best end-to-end system is ranked third after systems I and
pooling layers was employed for feature extraction as shown in III.
Figure 7. A single layer bidirectional LSTM and a single layer
TABLE IV. COMPARISION OF END TO END MODEL AND THE RECOGNITION
LSTM are used for the encoder and decoder, respectively. The SYSTEMS ON CROHME 2014 (%)
size of hidden states of the encoder and decoder is 256 and 512,
respectively. We used mini-batch stochastic gradient descent to Sym Seg
Measure Sym Exp
learn the parameters. The initial learning rate was set to 0.1. + Rel Tree
Method Seg Rec
The training process was stopped when the recognition rate on Rec
validation set stopped improving after 10 epochs. The system I 93.31 86.59 84.23 37.22
was implemented by using Torch and the Seq2seq-attn NMT II 76.63 66.97 60.31 15.01
system [14]. All the experiments were performed on a 4GB
III 98.42 93.91 94.26 62.68
Nvidia Tesla K20.
IV 85.52 76.64 70.78 18.97
C. Results V 88.23 78.45 61.38 18.97
The first experiment evaluated the performance of the end- VI 83.05 69.72 66.83 25.66
to-end systems trained on the CROHME training set,
G_CROHME1, and G_CROHME2. An OHME is recognized VII 89.43 76.53 71.77 26.06
correctly in terms of expression level if all of its symbols, End-to-end N/A N/A N/A 35.19
relations and its structure are recognized correctly. For
measurement, we use expression recognition rate which counts Finally, we evaluate the end-to-end system by structure
OHMEs recognized at the expression level over all the testing
recognition rate. Structure recognition rate is calculated by the
OHMEs. The training process is shown in Figure 6(b). Table II
percent of OHMEs whose structure is recognized correctly
shows the recognition rate on validation and testing sets by
using different training sets. The recognition rates on both irrespective of symbol labels. For example, the two OHMEs
validation and testing set increase when the number of training (x2 + 1 and x3 + 7) share the same structure. Table V shows the
patterns increases. structure recognition rates of the end-to-end systems trained
on the CROHME training set, G_CROHME1, and
TABLE II. PERFORMANCE OF END TO END SYSTEM ON DIFFERENT G_CROHME2. It shows that the end-to-end systems can learn
TRAINING SETS. well the structures of OHMEs. If we want to improve the
CROHME G_CROHME G_CROHME expression recognition rates of the end-to-end systems, the
Rec. rate
training set 1 2 remaining problem is how to improve the symbol recognition
Validation(%) 17.16 19.55 21.64 inside the end-to-end systems.
Testing(%) 18.97 21.10 26.27 TABLE V. STRUCTURE RECOGNITION RATE OF END-TO-END SYSTEMS
WITH DISTORTION ON TRAINING SET.

One of techniques to prevent over-fitting and improve CROHME G_CROHME G_CROHME

Rec. rate
training set 1 2
generalization of neural models is to use distortions at the
beginning of every epoch. In this experiment, we employed the Testing(%) 51.52 58.22 56.69
global distortion model described in Section III.B and the same
data in Table II. The training process is shown in Figure 6(a). Figure 8 shows examples recognized correctly and
The results are shown in Table III. Similarly, the recognition incorrectly by the end-to-end system trained on G_CROHME2.
rates increase when the number of training patterns increases.

TABLE III. PERFORMANCE OF END TO END SYSTEM WITH DISTORTION ON

TRAINING.

CROHME G_CROHME G_CROHME

Rec. rate
training set 1 2 ୬ ୬
ͳ
ܾ
Validation(%) 23.25 30.04 30.10 െ ሺ െ Ͳሻ ෍ š ୬ ൌ ෍ ›୬
ξʹ ξʹ ୧ୀଵ ୧ୀଵ
Testing(%) 28.09 34.99 35.19
a). Correctly recognition

Table IV shows our best recognition result and the results

of the systems which participated in the CROHME 2014
competition. The four factors are measured in the evaluation,
namely, Sym Seg as symbol segmentation rate, Sym Seg + Rec
as symbol segmentation and recognition rate, Rel Tree as rate ଶ ଶ
݀ଵ െ ʹ ܽଶ
ൌට ଡ଼ ൅ ୥
of structural analysis (termed “relation tree”), and Exp Rec as ݀ଵ ݀ଶ ൅ ʹ
expression recognition rate. The end-to-end system produces b) Incorrectly recognition
Fig. 8. Examples recognized correctly and incorrectly by our system.

900
 V. CONCLUSION [5] A. M. Awal, H. Mouchère, and C. Viard-Gaudin, A global learning
approach for an online handwritten mathematical expression recognition
In this paper, we have presented the end-to-end system for system, Pattern Recognition Letters, 2014, pp. 68–77.
recognizing OHMEs. We proposed a combination of the local [6] F. Alvaro, J. Sánchez, and J. Benedí, Recognition of On-line
and global distortion models for patterns generation. The Handwritten Mathematical Expressions Using 2D Stochastic Context-
efficiencies of the proposed local and global distortion models Free Grammars and Hidden Markov Models, Pattern Recognition
are demonstrated through the experiments. The recognition rate Letters, pp. 58-67, 2014.
is improved when we increase the number of training patterns. [7] A.D. Le and M. Nakagawa, A system for recognizing online
handwritten mathematical expressions by using improved structural
It achieves 28.09%, 34.99% and 35.19% by using distortion on analysis, International Journal of Document Analysis and Recognition, ,
the CROHME training set, G_CROHME1, and G_CROHME2, vol. 19, pp 305–319, 2016.
respectively. It shows that the end-to-end system is a potential [8] L Hu, R Zanibbi, MST-based visual parsing of online handwritten
system to compare with existing systems of OHME recognition. mathematical expressions, 15th International Conference on Frontiers in
Handwriting Recognition, 2016, pp. 337-342.
There still remain problems to improve the expression [9] T. Zhang, H. Mouchere, C. Viard-Gaudin: Using BLSTM for
recognition rate of the end-to-end system as follows. First, we interpretation of 2-D languages. Case of handwritten mathematical
should generate more OHMEs whose structures are more expressions. Document Numérique 19(2-3): 135-157 (2016).
varied and employ a larger memory GPU for training. Then, [10] Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine
we should improve the symbol recognition inside the end-to- translation by jointly learning to align and translate. arXiv preprint
end system by employing tree-structured LSTM [15] or arXiv:1409.0473.
decomposable attention model [16]. [11] Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.;
Zemel, R.; and Bengio, Y. 2015. Show, attend and tell: Neural image
ACKNOWLEDGMENT caption generation with visual attention. In Proceedings of The 32nd
International Conference on Machine Learning, 2048̽2057.
This research has been supported by JSPS fellowship under the [12] Y. Deng, A. Kanervisto, and A. M. Rush, What You Get Is What You
number 15J08654. See: A Visual Markup Decompiler, arXiv preprint
htp://arxiv.org/pdf/1609.04938v1.pdf
REFERENCES [13] B. Chen, B. Zhu and M. Nakagawa: Training of an On-line Handwritten
Japanese Character Recognizer by Artificial Patterns, Pattern
Recognition Letters, Vol. 35, No. 1, pp.178-185.
[1] K. Chan and D. Yeung, Mathematical Expression Recognition: A
Survey, International Journal of Document Analysis and Recognition, [14] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M.Rush, OpenNMT:
pp. 3-15, 2000 Open-Source Toolkit for Neural Machine Translation, ArXiv e-prints,
eprint = {1701.02810}.K. S. Tai, R. Socher, and C. Manning,
[2] R. Zanibbi and D. Blostein, Recognition and Retrieval of Mathematical Improved semantic representations from tree-structured long short-
Expressions, International Journal of Document Analysis and term memory networks. arXiv preprint arXiv:1503.00075, 2015.
Recognition , pp.331-357, 2012.
[15] K. S. Tai, R. Socher, and C. Manning, Improved semantic
[3] H. Mouchere, C. Viard-Gaudin, R. Zanibbi, and U. Garain, ICFHR 2014 representations from tree-structured long short-term memory
competition on recognition of on-line handwritten mathematical networks. arXiv preprint arXiv:1503.00075, 2015.
expressions (CROHME 2014). Proc. Int'l Conf. Frontiers in Handwriting
Recognition, pp. 791-796, 2014. [16] A. P. Parikh, O. Tackstrom, D. Das, J. Uszkoreit, A decomposable
attention model for natural language inference. arXiv preprint
[4] S. MacLean and G. Labahn, A new approach for recognizing arXiv:1606.01933 , 2016.
handwritten mathematics using relational grammars and fuzzy sets.
International Journal of Document Analysis and Recognition, vol. 16,
pp. 139-163, 2013.

901