ConvMath: A Convolutional Sequence Network for Mathematical Expression Recognition

Yan, Zuoyu; Zhang, Xiaode; Gao, Liangcai; Yuan, Ke; Tang, Zhi

doi:10.1109/icpr48806.2021.9412913

Cited by 18 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, these systems have evolved rapidly through deep learning. Usually, transformer-based approaches [2][3][4] have proven to outperform traditional statistical models [15,16] and convolutional neural networks [5,6,[17][18][19]. These neural networks are able to learn and recognize intricate patterns and features within images automatically, making them particularly well-suited for accurately extracting text with subscripts such as mathematical formulas from scanned documents or images [20].…”

Section: Mathematical Expression Recognitionmentioning

confidence: 99%

Investigating Models for the Transcription of Mathematical Formulas in Images

Feichter,

Schlippe

2024

Applied Sciences

View full text Add to dashboard Cite

The automated transcription of mathematical formulas represents a complex challenge that is of great importance for digital processing and comprehensibility of mathematical content. Consequently, our goal was to analyze state-of-the-art approaches for the transcription of printed mathematical formulas on images into spoken English text. We focused on two approaches: (1) The combination of mathematical expression recognition (MER) models and natural language processing (NLP) models to convert formula images first into LaTeX code and then into text, and (2) the direct conversion of formula images into text using vision-language (VL) models. Since no dataset with printed mathematical formulas and corresponding English transcriptions existed, we created a new dataset, Formula2Text, for fine-tuning and evaluating our systems. Our best system for (1) combines the MER model LaTeX-OCR and the NLP model BART-Base, achieving a translation error rate of 36.14% compared with our reference transcriptions. In the task of converting LaTeX code to text, BART-Base, T5-Base, and FLAN-T5-Base even outperformed ChatGPT, GPT-3.5 Turbo, and GPT-4. For (2), the best VL model, TrOCR, achieves a translation error rate of 42.09%. This demonstrates that VL models, predominantly employed for classical image captioning tasks, possess significant potential for the transcription of mathematical formulas in images.

show abstract

Section: Mathematical Expression Recognitionmentioning

confidence: 99%

Investigating Models for the Transcription of Mathematical Formulas in Images

Feichter,

Schlippe

2024

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…However, it does not solve the problem with variations in writing styles. Yan et al [22] developed ConvMath, a printed MER system based entirely on convolutions. They introduced a convolutional decoder to better detect the 2D relation of MEs.…”

Section: Related Workmentioning

confidence: 99%

MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition

Schmitt-Koopmann,

Huang,

Hutter

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Printed mathematical expression recognition (MER) models are usually trained and tested using LaTeX-generated mathematical expressions (MEs) as input and the LaTeX source code as ground truth. As the same ME can be generated by various different LaTeX source codes, this leads to unwanted variations in the ground truth data that bias test performance results and hinder efficient learning. In addition, the use of only one font to generate the MEs heavily limits the generalization of the reported results to realistic scenarios. We propose a data-centric approach to overcome this problem, and present convincing experimental results: Our main contribution is an enhanced LaTeX normalization to map any LaTeX ME to a canonical form. Based on this process, we developed an improved version of the benchmark dataset im2latex-100k, featuring 30 fonts instead of one. Second, we introduce the real-world dataset realFormula, with MEs extracted from papers. Third, we developed a MER model, MathNet, based on a convolutional vision transformer, with superior results on all four test sets (im2latex-100k, im2latexv2, realFormula, and InftyMDB-1), outperforming the previous state of the art by up to 88.3%.

show abstract

“…The MFR task is a specialized field aimed at automatically converting formula images into structured formula descriptions after locating them [71]. The MFR task is crucial for knowledge engineering and scientific document recognition.…”

Section: Extraction Of Mathematical Formula Informationmentioning

confidence: 99%

Construction of Remote Sensing Indices Knowledge Graph (RSIKG) Based on Semantic Hierarchical Graph

Wang,

Shi,

2023

Remote Sensing

View full text Add to dashboard Cite

Remote sensing indices are widely used in various fields of geoscience research. However, there are limits to how effectively the knowledge of indices can be managed or analyzed. One of the main problems is the lack of ontology models and research on indices, which makes it difficult to acquire and update knowledge in this area. Additionally, there is a lack of techniques to analyze the mathematical semantics of indices, making it difficult to directly manage and analyze their mathematical semantics. This study utilizes an ontology and mathematical semantics integration method to offer a novel knowledge graph for a remote sensing index knowledge graph (RSIKG) so as to address these issues. The proposed semantic hierarchical graph structure represents the indices of knowledge with an entity-relationship layer and a mathematical semantic layer. Specifically, ontologies in the entity-relationship layer are constructed to model concepts and relationships among indices. In the mathematical semantics layer, index formulas are represented using mathematical semantic graphs. A method for calculating similarity for index formulas is also proposed. The article describes the entire process of building RSIKG, including the extraction, storage, analysis, and inference of remote sensing index knowledge. Experiments provided in this article demonstrate the intuitive and practical nature of RSIKG for analyzing indices knowledge. Overall, the proposed methods can be useful for knowledge queries and the analysis of indices. And the present study lays the groundwork for future research on analysis techniques and knowledge processing related to remote sensing indices.

show abstract

ConvMath: A Convolutional Sequence Network for Mathematical Expression Recognition

Cited by 18 publications

References 19 publications

Investigating Models for the Transcription of Mathematical Formulas in Images

Investigating Models for the Transcription of Mathematical Formulas in Images

MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition

Construction of Remote Sensing Indices Knowledge Graph (RSIKG) Based on Semantic Hierarchical Graph

Contact Info

Product

Resources

About