PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks

Yu, Wenwen; Lu, Ning; Qi, Xianbiao; Gong, Ping; Xiao, Rong

doi:10.48550/arxiv.2004.07464

Cited by 4 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To encode the semantic entity in VRDs, Yu et al (2020b) and replace BiLSTM (Bi-directional Long Short-Term Memory) used by Liu et al (2019a) with BERT (Devlin et al, 2018) or RoBERTa (Liu et al, 2019b). Xu et al (2020b) propose LayoutLM, which adds the 2-D position embedding into language model based on BERT and pretrain their language model on large-scale scanned document images with more visually-related loss function.…”

Section: Related Workmentioning

confidence: 99%

“…While encoding VRDs, previous works take entity labeling task as sequence labeling and reimplement the named entity recognition (NER) framework (Lample et al, 2016) but ignore layout information. Then, many works introduce a GCNbased module to encode layout information and combine textual and visual information together (Liu et al, 2019a;Yu et al, 2020b;Carbonell et al, 2021). In the GCN module, Liu et al (2019a); Yu et al (2020b) take layout features between entity b i and b j as edge embedding to up-date entity representation while prune irrelevant nodes in graph according to same x-axis or y-axis coordinates to get the adjacency matrix.…”

Section: Related Workmentioning

confidence: 99%

“…These documents contain abundant layout information which helps us to understand the content while texts alone are not enough. In recent years, many works focus on how to extract key information from VRDs based on the results of OCR (Optical Character Recognition), which recognizes bounding boxes and texts within the boxes (Liu et al, 2019a;Yu et al, 2020b). Each bounding box contains 1) a group of words that belong together from a semantic and spatial standpoint and 2) visual features such as layout, tabular structure and font size of the boxes in the document.…”

Section: Introductionmentioning

confidence: 99%

“…Key information extraction (KIE) is such a task to analyze visually rich documents, which usually contains two steps, entity labeling and entity relation extraction. Similar to named entity recognition (NER) and relation extraction in the traditional natural language processing (NLP), entity labeling aims to assign predefined labels to the semantic entities in VRDs (G. Jaume and Thiran, 2019; Liu et al, 2019a;Yu et al, 2020b), and entity relation extraction 3 predicts relations between these semantic entities. Compared with NER and relation extraction in the traditional NLP, KIE from VRDs is a more challenging task.…”

Section: Introductionmentioning

confidence: 99%

“…• At the document encoder layer, we utilize graph convolutional networks (GCN) to combine textual and visual information in VRDs by mapping layout information into graph edge representation between entities (Liu et al, 2019a;Yu et al, 2020b).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Entity Relation Extraction as Dependency Parsing in Visually Rich Documents

Zhang¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Previous works on key information extraction from visually rich documents (VRDs) mainly focus on labeling the text within each bounding box (i.e., semantic entity), while the relations in-between are largely unexplored. In this paper, we adapt the popular dependency parsing model, the biaffine parser, to this entity relation extraction task. Being different from the original dependency parsing model which recognizes dependency relations between words, we identify relations between groups of words with layout information instead. We have compared different representations of the semantic entity, different VRD encoders, and different relation decoders. For the model training, we explore multi-task learning to combine entity labeling and relation extraction tasks; and for the evaluation, we conduct experiments on different datasets with filtering and augmentation. The results demonstrate that our proposed model achieves 65.96% F1 score on the FUNSD dataset. As for the realworld application, our model has been applied to the in-house customs data, achieving reliable performance in the production setting. * Corresponding author. The author's contributions were carried out while at Alibaba Group. His current affiliation is Vipshop (China) Co., Ltd.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Entity Relation Extraction as Dependency Parsing in Visually Rich Documents

Zhang¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding

Xu¹,

Xu²,

Lv³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

167

178

View full text Add to dashboard Cite

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-ofthe-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 → 0.8420), CORD (0.9493 → 0.9601), SROIE (0.9524 → 0.9781), Kleister-NDA (0.8340 → 0.8520), RVL-CDIP (0.9443 → 0.9564), and DocVQA (0.7295 → 0.8672). We made our model and code publicly available at https://aka.ms /layoutlmv2.

show abstract