Online Handwritten Mathematical Expression Recognition and Applications: A Survey

Abstract

Handwritten mathematical expressions are an essential part of many domains, including education, engineering, and science. The pervasive availability of computationally powerful touch-screen devices, similar to the recent emergence of deep neural networks as high-quality sequence recognition models, result in the widespread adoption of online recognition of handwritten mathematical expressions. Also, a deeper study and improvement of such technologies is necessary to address the current challenges posed by the extensive usage of distance learning, and remote work due to the world pandemic. This paper delineates the stateof-the-art recognition methods along with the user's experience in pen-centric applications for operating with handwritten mathematical expressions. Recognition methods have been categorized into classes, with a description of their merits and limitations. Particular attention is paid to end-to-end approaches based on encoder-decoder architecture and multi-modal input. Evaluation protocols and open benchmark datasets are considered as well as the comparison of the recognition performance, based on open competition results. The use of handwritten math recognition is illustrated by examples of applications for various fields and platforms. A distinctive part of the survey is that we also considered how UI design relies on the use of different recognition approaches, which is aimed at helping potential researchers improve the performance of the introduced approaches toward the best responses in practical applications. Finally, this paper presents the prospective survey of future research directions in handwritten mathematical expression recognition and their applications.

Doc2TeX: A Novel Approach To Convert Written Documents To LaTeX

“…These topics have been studied and summarized by various researchers over the years. The different types of grammars used were graph-grammar, definite-clause-grammar, relational grammar and Context-Free Grammar [6].…”

Two Decades of Bengali Handwritten Digit Recognition: A Survey

“…However, this area has a long way to go compared to the present state of mathematical expression recognition on other scripts [190]. Advanced techniques like stroke extraction [191] are found to be useful on the publicly available mathematical expression dataset CROHME [192].…”

A survey on handwritten mathematical expression recognition: The rise of encoder-decoder and GNN models

“…We also make reference to older research [6,7,8] because significant progress was made in earlier methods using structural approaches (i.e., grammar-based, tree-based, and graph-based) on publicly available datasets. Recently, Zhelezniakov et al [9] conducted a survey of online HME recognition considering systems, user interfaces, and applications from a broader perspective. In contrast, this survey provides an in-depth survey on recent HME recognition methods, and we believe that the two surveys are complementary.…”