MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction

Abstract

Visual Information Extraction (VIE) task aims to extract key information from multifarious document images (e.g., invoices and purchase receipts). Most previous methods treat the VIE task simply as a sequence labeling problem or classification problem, which requires models to carefully identify each kind of semantics by introducing multimodal features, such as font, color, layout. But simply introducing multimodal features couldn't work well when faced with numeric semantic categories or some ambiguous texts. To address this issue, in this paper we propose a novel key-value matching model based on a graph neural network for VIE (MatchVIE). Through key-value matching based on relevancy evaluation, the proposed MatchVIE can bypass the recognitions to various semantics, and simply focuses on the strong relevancy between entities. Besides, we introduce a simple but effective operation, Num2Vec, to tackle the instability of encoded values, which helps model converge more smoothly. Comprehensive experiments demonstrate that the proposed MatchVIE can significantly outperform previous methods. Notably, to the best of our knowledge, MatchVIE may be the first attempt to tackle the VIE task by modeling the relevancy between keys and values and it is a good complement to the existing methods.

Handling Big Tabular Data of Ict Supply Chains: A Multi-Task, Machine-Interpretable Approach

“…(3) (4) (5) The loss function may be described as Equation 6, which incorporates a TSR loss represented by , a CTC loss denoted by , and a hyper parameter to balance and since the suggested technique uses multi-task architecture, which means that all the parameters should be trained jointly. In our implementation, we employ cross entropy loss, which is defined as Equation 7, where is the prediction and represents the matching ground truth.…”

“…It is important to note that the majority of the current research [1,2,3] on CTC problems focuses on spreadsheet tables, which makes problem description considerably simpler because spreadsheets can store more meta-information and their default units are cells. Some research [4], [5] have attempted to extract entities and information from photos, which is similar to how we characterised the CTC issue. However, because they have not given attention to the tables in document images, their problem definitions cannot be solved simultaneously with the TSR problem.…”

DCMAI: A Dynamical Cross-Modal Alignment Interaction Framework for Document Key Information Extraction

“…However, these above methods only consider the text feature information, ignoring the auxiliary information such as visual and layout information in documents. Some GNN-based methods [4]- [6] applied multi-modal features of text segments as nodes to build the document graph and adopted the edge relationship of the graph network to inference the information relevance between neighboring nodes. Furthermore, TRIE [18] introduces an end-end network, which adopts a multi-modal context block to bridge the OCR and IE(information extraction) modules.…”

“…However, these above methods only consider the text feature information, ignoring the auxiliary information such as visual and layout information in documents. GNN-based methods [4]- [6] apply multi-modal features of text fragments as nodes and adopt the edge relationship of the GNN to evaluate the relations between each entity. However, these efforts rely on a defined set of entity categories/labels for each dataset, which prevents the application of the same DKIE model to other datasets.…”