Multimodal Mathematical Expressions Recognition: Case of Speech and Handwriting

Medjkoune, Sofiane; Mouchère, Harold; Petitrenaud, Simon; Viard-Gaudin, Christian

doi:10.1007/978-3-642-39330-3_9

Cited by 4 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main difficulties in HME voice input are associated with the identification of spatial relations (especially for right, subscript, and superscript), and the classification of characters with a similar way of writing. Medjkoune et al [113] suggested and explored various methods that reduce the uncertainty during MEs input that occurs in single-modality mode (Fig. 13).…”

Section: E Multi-modal Inputmentioning

confidence: 99%

Online Handwritten Mathematical Expression Recognition and Applications: A Survey

Zhelezniakov

Zaytsev²,

Radyvonenko³

2021

IEEE Access

View full text Add to dashboard Cite

Handwritten mathematical expressions are an essential part of many domains, including education, engineering, and science. The pervasive availability of computationally powerful touch-screen devices, similar to the recent emergence of deep neural networks as high-quality sequence recognition models, result in the widespread adoption of online recognition of handwritten mathematical expressions. Also, a deeper study and improvement of such technologies is necessary to address the current challenges posed by the extensive usage of distance learning, and remote work due to the world pandemic. This paper delineates the stateof-the-art recognition methods along with the user's experience in pen-centric applications for operating with handwritten mathematical expressions. Recognition methods have been categorized into classes, with a description of their merits and limitations. Particular attention is paid to end-to-end approaches based on encoder-decoder architecture and multi-modal input. Evaluation protocols and open benchmark datasets are considered as well as the comparison of the recognition performance, based on open competition results. The use of handwritten math recognition is illustrated by examples of applications for various fields and platforms. A distinctive part of the survey is that we also considered how UI design relies on the use of different recognition approaches, which is aimed at helping potential researchers improve the performance of the introduced approaches toward the best responses in practical applications. Finally, this paper presents the prospective survey of future research directions in handwritten mathematical expression recognition and their applications.

show abstract

Section: E Multi-modal Inputmentioning

confidence: 99%

Online Handwritten Mathematical Expression Recognition and Applications: A Survey

Zhelezniakov

Zaytsev²,

Radyvonenko³

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

Recent trends in mathematical expressions recognition: An LDA-based analysis

Sakshi

Kukreja

2023

Expert Systems with Applications

View full text Add to dashboard Cite

Text Alignment from Bimodal Mathematical Expression Sources

Medjkoune¹,

Mouchère²,

Viard-Gaudin³

et al. 2014

2014 14th International Conference on Frontiers in Handwriting Recognition

View full text Add to dashboard Cite

International audienceIn this paper we propose a new approach tomerge mathematical expression recognition results coming fromhandwriting and speech modalities. Using a bimodal descriptionof mathematical expressions allows taking advantage of thecomplementarities between both signals, and can disambiguatesituations were a single modality would not be clear enough. Tocombine the signals coming from both modalities, we propose torepresent them in the same space as a textual description. First,from the handwriting signal, we generate the Nbest mathematicalexpressions; each of them is next translated as different possiblestrings. From the audio signal, an automatic speech recognitionsystem provides a transcript, which is also available as a string.A string comparison algorithm is achieved to select the bestmathematical expressions. This bimodal system is evaluated onreal bimodal data from the HAMEX dataset and the results arecompared to a single modality (handwriting) based system

show abstract

Multimodal Mathematical Expressions Recognition: Case of Speech and Handwriting

Cited by 4 publications

References 12 publications

Online Handwritten Mathematical Expression Recognition and Applications: A Survey

Online Handwritten Mathematical Expression Recognition and Applications: A Survey

Recent trends in mathematical expressions recognition: An LDA-based analysis

Text Alignment from Bimodal Mathematical Expression Sources

Contact Info

Product

Resources

About