An End-to-End Formula Recognition Method Integrated Attention Mechanism

Abstract

Formula recognition is widely used in document intelligent processing, which can significantly shorten the time for mathematical formula input, but the accuracy of traditional methods could be higher. In order to solve the complexity of formula input, an end-to-end encoder-decoder framework with an attention mechanism is proposed that converts formulas in pictures into LaTeX sequences. The Vision Transformer (VIT) is employed as the encoder to convert the original input picture into a set of semantic vectors. Due to the two-dimensional nature of mathematical formula, in order to accurately capture the formula characters’ relative position and spatial characteristics, positional embedding is introduced to ensure the uniqueness of the character position. The decoder adopts the attention-based Transformer, in which the input vector is translated into the target LaTeX character. The model adopts joint codec training and Cross-Entropy as a loss function, which is evaluated on the im2latex-100k dataset and CROHME 2014. The experiment shows that BLEU reaches 92.11, MED is 0.90, and Exact Match(EM) is 0.62 on the im2latex-100k dataset. This paper’s contribution is to introduce machine translation to formula recognition and realize the end-to-end transformation from the trajectory point sequence of formula to latex sequence, providing a new idea of formula recognition based on deep learning.

Investigating Models for the Transcription of Mathematical Formulas in Images

“…In recent years, these systems have evolved rapidly through deep learning. Usually, transformer-based approaches [2][3][4] have proven to outperform traditional statistical models [15,16] and convolutional neural networks [5,6,[17][18][19]. These neural networks are able to learn and recognize intricate patterns and features within images automatically, making them particularly well-suited for accurately extracting text with subscripts such as mathematical formulas from scanned documents or images [20].…”

“…Figure 1 illustrates the two possible processes for transcribing mathematical formulas. In the first approach, the image of the formula is converted into LaTeX code using mathematical expression recognition (MER) [2][3][4][5][6]. This LaTeX code then serves as input for a natural language processing (NLP) model, which produces the transcription of the formula.…”

“…Retrieve a complete transcription that covers the whole formula, "sigma is equal to the square root of one divided by n times the sum from i equals one to n of xi minus mu squared"). In the automatic transcription of mathematical formulas, challenge 1 and challenge 2 are usually handled using MER models, which convert the image of the formula into LaTeX code [2][3][4][5][6]. As demonstrated in Figure 3, LaTeX code usually contains commands such as \lim, \sum, and \frac.…”

Advancing OCR Accuracy in Image-to-LaTeX Conversion—A Critical and Creative Exploration

“…Histogram equalization techniques, such as adaptive histogram equalization (AHE) and contrast-limited adaptive histogram equalization (CLAHE), have been widely used. Recent research has explored integrating deep learning models, such as U-Net and Pix2Pix networks, for adaptive contrast enhancement [25]. These approaches have demonstrated their effectiveness in handling varying illumination conditions and improving OCR performance.…”

“…Binarization and thresholding techniques are also critical in OCR preprocessing. Binarization converts grayscale or color images into binary representations, separating foreground characters from the background [25]. Various thresholding techniques, including global thresholding, local adaptive thresholding, and hybrid methods, have been proposed to address different image characteristics and challenges.…”