CoMER: Modeling Coverage for Transformer-Based Handwritten Mathematical Expression Recognition

Zhao, Wenqi; Gao, Liangcai

doi:10.1007/978-3-031-19815-1_23

Cited by 18 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Li et al [16] introduced a counting module, greatly enhancing model performance. Additionally, other strategies such as transformer-based decoders, as used in [30,31,32], have also achieved encouraging performance. Tree decoder methods treat Mathematical Expressions (ME) as a tree structure, a natural representation for the tree structure of ME in the HMER context [36].…”

Section: Related Work 21hmermentioning

confidence: 99%

DGNet: A Handwritten Mathematical Formula Recognition Network Based on Deformable Convolution and Global Context Attention

Wen,

Yin,

Liu

2024

Mobile Netw Appl

View full text Add to dashboard Cite

Handwritten Mathematical Expression Recognition (HMER) task aims to generate corresponding LATEX sequences from images of handwritten mathematical expressions. Currently, the encoder-decoder architecture has made significant progress in this task. However, the architecture based on the DenseNet encoder fails to adequately consider the unique features of handwritten mathematical expressions (HME) and the similarity between different characters. Additionally, the decoder, with its small receptive field during the decoding process, fails to effectively capture the spatial positional information of the targets, resulting in a lack of global contextual information during decoding.To address these issues, this paper proposes a neural network called DGNet based on deformable convolution and global contextual attention. Our network takes into full consideration the sparse nature of handwritten mathematical formulas and utilizes the properties of deformable convolution, allowing the convolution kernel to deform based on the content of the neighborhood. This enables our model to better adapt to geometric changes and other deformations in handwritten mathematical expressions. Simultaneously, we introduce GCAttention in optimizing the feature part to fully aggregate global contextual features of both position and channel.In experiments, our model achieved accuracies

show abstract

Section: Related Work 21hmermentioning

confidence: 99%

DGNet: A Handwritten Mathematical Formula Recognition Network Based on Deformable Convolution and Global Context Attention

Wen,

Yin,

Liu

2024

Mobile Netw Appl

View full text Add to dashboard Cite

show abstract

“…Zhao e Gao ainda propuseram dois mecanismos de cobertura adicionais ao BTTR, além de um método de aumento de dados, a fim de melhorar a performance e definiram um novo algoritmo: Coverage information in the transforMER decoder (CoMER) [Zhao and Gao 2022].…”

Section: Bttr E Comerunclassified

“…Para implementar essa ideia, pode-se explorar o reconhecimento de caracteres/dígitos: uma técnica de visão computacional que reconhece dígitos e caracteres escritos à mão, entre outros símbolos matemáticos [Mouchère et al 2014]. No entanto, apesar de muitas pesquisas nessa área [Davis et al 2020, Vu et al 2021, Patel et al 2022, Zhao et al 2021, Yuan et al 2022a, Li et al 2022, Yuan et al 2022b, Zhao and Gao 2022, estudos anteriores sobre STIs para educac ¸ão matemática com base em entrada escrita à mão não experimentaram com algoritmos de última gerac ¸ão para reconhecimento de dígitos [Davis et al 2020], ou são limitados a problemas de álgebra [Patel et al 2022]. Para preencher essa lacuna, este artigo apresenta um estudo experimental comparando quatro algoritmos de última gerac ¸ão para reconhecimento de dígitos no contexto de problemas matemáticos de adic ¸ão e subtrac ¸ão.…”

Section: Introduc ¸ãOunclassified

Algoritmos de Reconhecimento de Dígitos para Integração de Equações Manuscritas em Sistemas Tutores Inteligentes

Chevtchenko,

Rodrigues,

Rosa

et al. 2023

Anais Do XXXIV Simpósio Brasileiro De Informática Na Educação (SBIE 2023)

View full text Add to dashboard Cite

Sistemas Tutores Inteligente (STIs) têm sido amplamente utilizados para auxiliar no aprendizado de matemática. No entanto, a diferença na forma de inserção de soluções nos STIs, que requer o uso de um teclado, em comparação com a prática padrão de escrever à mão, pode levar a problemas de usabilidade e prejudicar a aprendizagem. Para superar essa limitação, pesquisas recentes têm explorado o reconhecimento de caracteres escritos à mão em papel como entrada para os STIs. Porém, existe uma lacuna de conhecimento em relação ao desempenho dos algoritmos de reconhecimento de dígitos avançados no contexto de operações matemáticas básicas. Este artigo compara quatro algoritmos de última geração para o reconhecimento de dígitos em problemas matemáticos de adição e subtração. Os resultados revelam que o algoritmo BTTR obteve o melhor desempenho em termos de acurácia, enquanto o algoritmo SAN apresentou um bom equilíbrio entre acurácia e velocidade de reconhecimento. Essas descobertas são relevantes para pesquisadores e desenvolvedores ao selecionar os algoritmos mais adequados para o desenvolvimento de STIs baseados em entrada escrita à mão.

show abstract

SET, SORT! A Novel Sub-stroke Level Transformers for Offline Handwriting to Online Conversion

Mohamed Moussa,

Lelore,

Mouchère

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We present a novel sub-stroke level transformer approach to convert offline images of handwriting to online. We start by extracting sub-strokes from the offline images by inferring a skeleton with a CNN and applying a basic cutting algorithm. We introduce sub-stroke embeddings by encoding the sub-stroke point sequence with a Sub-stroke Encoding Transformer (SET). The embeddings are then fed to the Substrokes ORdering Transformer (SORT) which predicts the discrete substrokes ordering and the pen state. By constraining the Transformer input and output to the inferred sub-strokes, the recovered online is highly precise. We evaluate our method on Latin words from the IRONOFF dataset and on maths expressions from CROHME dataset. We measure the performance with two criteria: fidelity with Dynamic Time Warping (DTW) and semantic coherence using recognition rate. Our method outperforms the state-of-the-art in both datasets, achieving a word recognition rate of 81.06% and a 2.41 DTW on IRONOFF and an expression recognition rate of 62.00% and a DTW of 13.93 on CROHME 2019. This work constitutes an important milestone toward full offline document conversion to online.

show abstract

CoMER: Modeling Coverage for Transformer-Based Handwritten Mathematical Expression Recognition

Cited by 18 publications

References 25 publications

DGNet: A Handwritten Mathematical Formula Recognition Network Based on Deformable Convolution and Global Context Attention

DGNet: A Handwritten Mathematical Formula Recognition Network Based on Deformable Convolution and Global Context Attention

Algoritmos de Reconhecimento de Dígitos para Integração de Equações Manuscritas em Sistemas Tutores Inteligentes

SET, SORT! A Novel Sub-stroke Level Transformers for Offline Handwriting to Online Conversion

Contact Info

Product

Resources

About