Watch Your Strokes: Improving Handwritten Text Recognition with Deformable Convolutions

Abstract

Handwritten Text Recognition (HTR) in free-layout pages is a valuable yet challenging task which aims to automatically understand handwritten texts. State-of-the-art approaches in this field usually encode input images with Convolutional Neural Networks, whose kernels are typically defined on a fixed grid and focus on all input pixels independently. However, this is in contrast with the sparse nature of handwritten pages, in which only pixels representing the ink of the writing are useful for the recognition task. Furthermore, the standard convolution operator is not explicitly designed to take into account the great variability in shape, scale, and orientation of handwritten characters. To overcome these limitations, we investigate the use of deformable convolutions for handwriting recognition. The kernel of this type of convolution deforms according to the content of the neighborhood, and can therefore be more adaptable to geometric variations and other deformations of the text. Experiments conducted on the IAM and RIMES datasets demonstrate that the use of deformable convolutions is a promising direction for the design of novel architectures for handwritten text recognition.

DGNet: A Handwritten Mathematical Formula Recognition Network Based on Deformable Convolution and Global Context Attention

“…This evolution in technology highlights a deep understanding of model adaptability and non-local structural modeling capabilities, providing robust support for research and applications in the field of computer vision. Iulian Cojocaru et al [40] introduced deformable convolution into handwritten text recognition, recognizing that standard convolutional operators do not explicitly consider the significant variability in the shape, proportion, and orientation of handwritten characters. To overcome these limitations, the authors studied deformable convolution and ultimately demonstrated its effectiveness in handwritten recognition.…”

Boosting modern and historical handwritten text recognition with deformable convolutions

“…This strategy was the standard one [6,19,35] until simpler alternatives to MDLSTM-RNNs were proposed [38,43]. These consist of a ConvNet to extract a sequence of feature vectors from the text image and 1D-LSTMs to output character probabilities for the CTC decoding and became commonly used as a backbone for HTR systems [8,13,16,36,50] due to its performance and faster training compared to MDLSTMs.…”

“…As a consequence, deformable convolutions can adapt to the input geometric variations and part deformations, making them potentially more suitable for dealing with HTR images compared to standard convolutions. This kind of convolution has been originally proposed to tackle the object recognition task and, to the best of our knowledge, its usage in HTR has been explored only in our preliminary work [16], where we claim that its kernel adaptability (see Figure 1) can help to improve the efficiency and the performance in the task. In this work, we deepen our analysis and extend it to HTR on historical manuscripts.…”