Perceiving Stroke-Semantic Context: Hierarchical Contrastive Learning for Robust Scene Text Recognition

Abstract

We introduce Perceiving Stroke-Semantic Context (PerSec), a new approach to self-supervised representation learning tailored for Scene Text Recognition (STR) task. Considering scene text images carry both visual and semantic properties, we equip our PerSec with dual context perceivers which can contrast and learn latent representations from low-level stroke and high-level semantic contextual spaces simultaneously via hierarchical contrastive learning on unlabeled text image data. Experiments in un- and semi-supervised learning settings on STR benchmarks demonstrate our proposed framework can yield a more robust representation for both CTC-based and attention-based decoders than other contrastive learning methods. To fully investigate the potential of our method, we also collect a dataset of 100 million unlabeled text images, named UTI-100M, covering 5 scenes and 4 languages. By leveraging hundred-million-level unlabeled data, our PerSec shows significant performance improvement when fine-tuning the learned representation on the labeled data. Furthermore, we observe that the representation learned by PerSec presents great generalization, especially under few labeled data scenes.

MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining

“…Our small model achieves the best performance over the previous counterparts with the similar model size. Specifically, our method achieves better accuracy than PerSec [30] which pretrained with 100 million real data, while we only use 4.2 million real data for pretraining. ABINet [12] and its extension ConCLR [54], performing similarly to our approach with small ViT, use an explicit pretrained masked language model to correct the predicted result iteratively, which is complementary to our approach and may benefit our approach.…”

Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition

“…The proposed framework was effective with both CTC decoders and attention decoders. PerSec [34] also introduced contrastive learning for text recognition. It presented hierarchical contrastive learning that drove each element of features at high and low levels.…”

“…Data augmentation For effective contrastive learning, the data augmentation plays an essential role in the whole framework. Following [34], besides the augmentation operations used in SeqCLR, more image augmentations including color jitter and gray are employed to enhance the representation quality of the pre-trained model.…”

“…Handwritten Text Recognition Benchmarks We verify our text recognition models on two handwritten text recognition benchmarks: CVL [26] and IAM [38]. Following SeqCLR [1] and Per-Sec [34] [36] with a cosine learning rate scheduler, and train the models for 3 epochs. The training hyperparameters are: the batch size as 1,024, base learning rate as 1.5e-4, weight decay as 0.05, 𝛽 1 = 0.9, 𝛽 2 = 0.95, warm-up for 5,000 steps.…”

“…SeqCLR [1] proposed a sequence-to-sequence contrastive learning framework for text recognition. PerSec [34] introduced a hierarchical contrastive learning method for text recognition. They are both based on contrastive learning and mainly focus on learning the discrimination of text images, as shown in Fig.…”