Recognition of Online Handwritten Math Symbols Using Deep Neural Networks

Handwritten mathematical expressions are an essential part of many domains, including education, engineering, and science. The pervasive availability of computationally powerful touch-screen devices, similar to the recent emergence of deep neural networks as high-quality sequence recognition models, result in the widespread adoption of online recognition of handwritten mathematical expressions. Also, a deeper study and improvement of such technologies is necessary to address the current challenges posed by the extensive usage of distance learning, and remote work due to the world pandemic. This paper delineates the stateof-the-art recognition methods along with the user's experience in pen-centric applications for operating with handwritten mathematical expressions. Recognition methods have been categorized into classes, with a description of their merits and limitations. Particular attention is paid to end-to-end approaches based on encoder-decoder architecture and multi-modal input. Evaluation protocols and open benchmark datasets are considered as well as the comparison of the recognition performance, based on open competition results. The use of handwritten math recognition is illustrated by examples of applications for various fields and platforms. A distinctive part of the survey is that we also considered how UI design relies on the use of different recognition approaches, which is aimed at helping potential researchers improve the performance of the introduced approaches toward the best responses in practical applications. Finally, this paper presents the prospective survey of future research directions in handwritten mathematical expression recognition and their applications.

Section: A Symbol Segmentation and Classificationmentioning

confidence: 99%

Online Handwritten Mathematical Expression Recognition and Applications: A Survey

Zhelezniakov

Zaytsev²,

Radyvonenko³

2021

“…From the perspective of model and feature selection, CNN [15] is superior to traditional machine learning methods (SVM-RBF [18], AdaBoost [11], GLVQ [13]), which is higher by 2% to 3%. However, it is lower than the method of combining online, such as MLP + RNN [13], [20], CNN + LSTM [19], RNN [17], [20]. Each model utilizes RNN to capture long-term context dependencies of sequential data, but its temporal dynamic behavior with the delay brings serious shortcoming.…”

Section: Comparison With Existing Algorithmsmentioning

confidence: 99%

Multi-Feature Learning by Joint Training for Handwritten Formula Symbol Recognition

Fang

Zhang

2020

Given the similarity of handwritten formula symbols and various handwriting styles, this paper proposes a squeeze-extracted multi-feature convolution neural network (SE-MCNN) to improve the recognition rate of handwritten formula symbols. The system proposed in this paper integrates the eightdirectional feature of the original sequence in the convolutional layer, which significantly compensates for the lost dynamic trajectory information in the handwritten formula symbol. Meanwhile, the joint loss is constructed to improve the discriminability of features in the way of supervised learning, which enlarges the inter-class difference and decreases inner-class similarity. The standard mathematical formula symbol library provided by the Competition Organization on Recognition of Online Handwritten Mathematical Expression (CROHME) is used to verify the effectiveness of the proposed algorithm. Experiments show that the proposed SE-MCNN approach outperforms the state-of-the-art methods even at the condition of without using the data augmentation. INDEX TERMS Artificial neural network, multi-feature, joint training, discriminative feature.

“…For text line recognition, Segmentation based methods, which attempt to split text line into characters at their true boundaries and label the split characters by using Hidden Markov Model [8] or CNN [9,10]. Segmentation free method show their advantages in the problems of the sequence to sequence, such as handwritten recognition [11], speech recognition [12].…”

Section: Related Workmentioning

confidence: 99%

A Human-Inspired Recognition System for Pre-Modern Japanese Historical Documents

Clanuwat

Kitamoto

2019

Recognition of historical documents is a challenging problem due to the noised, damaged characters, and background. However, in Japanese historical documents, not only contains the mentioned problems, pre-modern Japanese characters were written in cursive and are connected. Therefore, character segmentation-based methods do not work well. This leads to the idea of creating a new recognition system. In this paper, we propose a human-inspired document reading system to recognize multiple lines of premodern Japanese historical documents. During the reading, people employ eyes movement to determine the start of a text line. Then, they move the eyes from the current character/word to the next character/word. They can also determine the end of a line or skip a figure to move to the next line. The eyes movement integrates with visual processing to operate the reading process in the brain. We employ attention-based encoder-decoder to implement this recognition system. First, the recognition system detects were to start a text line. Second, the system scans and recognize character by character until the text line is completed. Then, the system continues to detect the start of the next text line. This process is repeated until reading the whole document. As results, the system is successful to recognize multiple lines, connected and cursive characters without performing character/line segmentation. Besides, we also employ a coverage model which stores the history of eyes movement to predict the next movement more precisely. We tested our human-inspired recognition system on the pre-modern Japanese historical document provided by the PRMU Kuzushiji competition. The results of the experiments demonstrate the superiority and effectiveness of our proposed system by achieving Sequence Error Rate of 9.87% and 53.81% on level 2 and level 3 of the dataset, respectively. These results outperform to any other systems participated in the PRMU Kuzushiji competition. INDEX TERMS A human reading-inspired recognition system, recognition of pre-modern Japanese historical document, attention-based encoder-decoder, Kuzushiji.