Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

Abstract

Linguistic knowledge is of great benefit to scene text recognition. However, how to effectively model linguistic rules in end-to-end deep networks remains a research challenge. In this paper, we argue that the limited capacity of language models comes from: 1) implicitly language modeling; 2) unidirectional feature representation; and 3) language model with noise input. Correspondingly, we propose an autonomous, bidirectional and iterative ABINet for scene text recognition. Firstly, the autonomous suggests to block gradient flow between vision and language models to enforce explicitly language modeling. Secondly, a novel bidirectional cloze network (BCN) as the language model is proposed based on bidirectional feature representation. Thirdly, we propose an execution manner of iterative correction for language model which can effectively alleviate the impact of noise input. Additionally, based on the ensemble of iterative predictions, we propose a self-training method which can learn from unlabeled images effectively. Extensive experiments indicate that ABINet has superiority on lowquality images and achieves state-of-the-art results on several mainstream benchmarks. Besides, the ABINet trained with ensemble self-training shows promising improvement in realizing human-level recognition. Code is available at https://github.com/FangShancheng/ABINet.

Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study

“…Also, some works [21,58,82] attempt to construct text recognizers based on Transformer [64], which is thrived in the field of natural language processing, to robustly learn the representations for text images through self-attention modules. Recently, some researchers incorporated semantic knowledge into text recognizers to fully exploit the external language priors [20,54,75,88]. For example, SEED [54] utilizes text embedding guided by fastText [8] to initialize the attention-based decoder.…”

“…For example, SEED [54] utilizes text embedding guided by fastText [8] to initialize the attention-based decoder. ABINet [20] designs two autonomous branches to iteratively optimize the vision and language models. Overall, these general text recognition methods can be easily transferred to Chinese scenarios by simply replacing the alphabet or language priors.…”

OCR-IDL: OCR Annotations for Industry Document Library Dataset

“…More specifically, they obtain OCR annotations from an open source OCR engine Tesseract [45] for 5 Million documents from IIT-CDIP [25] dataset. With the introduction of pre-training strategy and advances in modern OCR engine [1,12,20,28,34], many contemporary approaches [7,2,53] have utilized even more data to advance the Document Intelligence field.…”

Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features

“…They typically consist of a visual feature extractor, abstracting the image patch, and a character sequence generator, responsible for character decoding. Despite wide explorations to find better visual feature extractors and character sequence generators, existing methods still suffer from challenging environments: occlusion, (a) STR methods w/ an LM [6,28] (b) Semantic-MATRN and [3] (c) Visual-MATRN and [25] (d) MATRN blurs, distortions, and other artifacts [2,3].…”

“…Once identifying a seed character se-quence, a bi-directional Transformer encoder re-estimates the character at each position. Based on SRN, Fang et al [6] improve the iterative refinement stages by explicitly dividing a vision model (VM) and an LM by blocking gradient flows and employing a bi-directional LM pre-trained on unlabeled text datasets. These methods incorporating semantic knowledge of LMs provide breakthroughs in recognizing challenging examples with ambiguous visual clues.…”