Multimodal grid features and cell pointers for scene text visual question answering

Gómez, Lluís; Biten, Ali Furkan; Tito, Rubèn; Mafla, Andrés; Rusiñol, Marçal; Valveny, Ernest; Karatzas, Dìmosthenis

doi:10.1016/j.patrec.2021.06.026

Cited by 17 publications

(4 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Han et al 26 presents the (LaAP‐Net), a model which not only guesses answers to questions together with generation of the bounding boxes as proof to support the predicted answer. An innovative STVQA model that uses an attention method to mutually reason about textual and visual modalities was developed by Gomez et al 27 By attending to multi‐modal grid features, their model achieves a comprehensive understanding of the scene, enhancing performance in the task. The Cascade Reasoning Network (CRN), 28 which combines a progressive attention module (PAM) with a multimodal reasoning graph (MRG) module, was suggested by Liu et al 28 The PAM performs stepwise encoding to fuse multimodal information, leveraging previous attention results to guide subsequent fusion steps.…”

Section: Related Workmentioning

confidence: 99%

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Agrawal,

Jalal,

Sharma

2024

Computational Intelligence

View full text Add to dashboard Cite

Visual question answering (VQA) is a challenging task in computer vision. Recently, there has been a growing interest in text‐based VQA tasks, emphasizing the important role of textual information for better understanding of images. Effectively utilizing text information within the image is crucial for achieving success in this task. However, existing approaches often overlook the contextual information and neglect to utilize the relationships between scene‐text tokens and image objects. They simply incorporate the scene‐text tokens mined from the image into the VQA model without considering these important factors. In this paper, the proposed model initially analyzes the image to extract text and identify scene objects. It then comprehends the question and mines relationships among the question, OCRed text, and scene objects, ultimately generating an answer through relational reasoning by conducting semantic and positional attention. Our decoder with attention map loss enables prediction of complex answers and handles dynamic vocabularies, reducing decoding space. It outperforms softmax‐based cross entropy loss in accuracy and efficiency by accommodating varying vocabulary sizes. We evaluated our model's performance on the TextVQA dataset and achieved an accuracy of 53.91% on the validation set and 53.98% on the test set. Moreover, on the ST‐VQA dataset, our model obtained ANLS scores of 0.699 on the validation set and 0.692 on the test set.

show abstract

Section: Related Workmentioning

confidence: 99%

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Agrawal,

Jalal,

Sharma

2024

Computational Intelligence

View full text Add to dashboard Cite

show abstract

“…For the M+N targets, the above method is used to obtain the corresponding locational location manifestation X loc 2R (M+N)×(M+N)×4 . Furthermore, X loc is passed through a two-layer fully connected network to obtain the final locational relation knowledge manifestation R loc 2R (M +N)×(M+N) , as shown in (3).…”

Section: Locational Collaboration Knowledge Manifestation With Inter-...mentioning

confidence: 99%

“…Literature [2] and Literature [3] proposed integrating textual content into VQA, forming the task of TextVQA (Text Visual Question Answering) and ST-VQA (Scenario Text Visual Question Answering) respectively, along with the construction of benchmark datasets. Fig 2 illustrates an example of the ST-VQA task, where the questions are related to the scenario text in the image, requiring the model to establish unified collaborations between the question, visual targets, and scenario text to generate correct answers.…”

Section: Introductionmentioning

confidence: 99%

Image to English translation and comprehension: INT2-VQA method based on inter-modality and intra-modality collaborations

Sheng

2023

PLoS ONE

View full text Add to dashboard Cite

Existing visual question answering methods typically concentrate only on visual targets in images, ignoring the key textual content in the images, thereby limiting the depth and accuracy of image content comprehension. Inspired by this, we pay attention to the task of text-based visual question answering, address the performance bottleneck issue caused by over-fitting risk in existing self-attention-based models, and propose a scenario text visual question answering method called INT2-VQA that fuses knowledge manifestation based on inter-modality and intra-modality collaborations. Specifically, we model the complementary priori knowledge of locational collaboration between visual targets and textual targets across modalities and the contextual semantical collaboration among textual word targets within a modality. Based on this, a universal knowledge-reinforced attention module is designed to achieve a unified encoding manifestation of both relations. Extensive ablation experiments, contrast experiments, and visual analyses demonstrate the effectiveness of the proposed method and prove its superiority over the other state-of-the-art methods.

show abstract

“…Secondly, we decide to use a commercial OCR engine, specifically Amazon Textract 3 , over Tesseract. It is because the performance of the OCR engines can significantly affect the model's performance which can be seen in fields that use OCR annotations, such as in fine-grained classification [29,30,31], in scenetext visual question answering [9,44,8,13], in document visual question answering (DocVQA) [50,33]. Apart from improving the annotation quality significantly, we want to level the differences between research groups and companies.…”

Section: Introductionmentioning

confidence: 99%

OCR-IDL: OCR Annotations for Industry Document Library Dataset

Biten¹,

Tito²,

Gómez³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Pretraining has proven successful in Document Intelligence tasks where deluge of documents are used to pretrain the models only later to be finetuned on downstream tasks. One of the problems of the pretraining approaches is the inconsistent usage of pretraining data with different OCR engines leading to incomparable results between models. In other words, it is not obvious whether the performance gain is coming from diverse usage of amount of data and distinct OCR engines or from the proposed models. To remedy the problem, we make public the OCR annotations for IDL documents using commercial OCR engine given their superior performance over open source OCR models. The contributed dataset (OCR-IDL) has an estimated monetary value over 20K US$. It is our hope that OCR-IDL can be a starting point for future works on Document Intelligence. All of our data and its collection process with the annotations can be found in https://github.com/furkanbiten/idl_data.

show abstract

Multimodal grid features and cell pointers for scene text visual question answering

Cited by 17 publications

References 3 publications

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Image to English translation and comprehension: INT2-VQA method based on inter-modality and intra-modality collaborations

OCR-IDL: OCR Annotations for Industry Document Library Dataset

Contact Info

Product

Resources

About