Multi-modal Text Recognition Networks: Interactive Enhancements Between Visual and Semantic Features

Na, Byeonghu; Kim, Yoon-Sik; Park, Sungrae

doi:10.1007/978-3-031-19815-1_26

Cited by 25 publications

(14 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Generally, language-augmented methods (i.e. , SRN [8], Vision-LAN [55], PARSeq [57], ABINet++ [104], LevOCR [56], MA-TRN [10] and MGP-STR) perform better than language-free methods, showing the significance of linguistic information. PTIE [51], which utilizes a transformer-only model with multiple patch resolutions, also achieves good results.…”

Section: Results On Standard Benchmarksmentioning

confidence: 99%

“…We conduct qualitative comparisons with two representative STR methods (i.e. ABINet [9] and MATRN [10]) on typical images from standard benchmarks. Fig.…”

Section: Qualitative Resultsmentioning

confidence: 99%

“…VisionLAN [55] proposes a visual reasoning module that simultaneously captures visual and language information by masking input images at the feature level. LevOCR [56] and MATRN [10] propose cross-modal transformer modules to explore the interactions between vision and language features for better recognition performances. PARSeq [57] learns an ensemble of internal language modules with shared architecture and weights using a transformer decoder, and it still retains the ability of iterative refinement like ABINet [9].…”

Section: Language-augmented Str Methodsmentioning

confidence: 99%

“…Generally, most works [9], [10], [57] mentioned above utilize a specific module to capture semantic information at the character level, while we manage to directly utilize multi-granularity semantic information (characters, subwords and even whole words) for STR in this paper.…”

Section: Language-augmented Str Methodsmentioning

confidence: 99%

“…ABINet [9] proposed Bidirectional Cloze Network (BCN) as the language model to learn bidirectional feature representation. MA-TRN [10] made a further step by introducing multi-modal feature enhancements with bi-directional fusions. These STR methods, which explicitly adopt an independent and separate language model to capture rich language prior, have proven to achieve considerable performance improvements.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multi-granularity Prediction for Scene Text Recognition

Wang

Cheng

Yao

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Due to the enormous technical challenges and wide range of applications, scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this tough problem, numerous innovative methods have been successively proposed, and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet functionally powerful vision STR model, which is built upon ViT and a tailored Adaptive Addressing and Aggregation (A 3 ) module. It already outperforms most previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. To produce the final recognition results, two strategies for effectively fusing the multi-granularity predictions are devised: the first strategy, called Confidence-based Fusion Strategy (CFS), employs a straightforward rule based on the confidence scores of the multi-granularity predictions, while the second one, called Learnable Fusion Strategy (LFS), is realized with a trainable module to directly measure the cross-modal similarities between text (predicted words) and images, akin to CLIP. The resultant algorithm (termed MGP-STR) is able to push the performance envelope of STR to an even higher level. Specifically, MGP-STR with LFS achieves an average recognition accuracy of 94% on standard benchmarks for scene text recognition (such as IIIT 5K-word, ICDAR 2015, SVT and CUTE). Moreover, it also achieves state-of-the-art results on widely-used handwritten benchmarks (IAM, CVL and RIMES) as well as more challenging scene text datasets (ArT, COCO-Text and Uber-Text), demonstrating the generality of the proposed MGP-STR algorithm. The source code and models will be available at: https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR.

show abstract

Section: Results On Standard Benchmarksmentioning

confidence: 99%

“…We conduct qualitative comparisons with two representative STR methods (i.e. ABINet [9] and MATRN [10]) on typical images from standard benchmarks. Fig.…”

Section: Qualitative Resultsmentioning

confidence: 99%