LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding

Abstract

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-ofthe-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 → 0.8420), CORD (0.9493 → 0.9601), SROIE (0.9524 → 0.9781), Kleister-NDA (0.8340 → 0.8520), RVL-CDIP (0.9443 → 0.9564), and DocVQA (0.7295 → 0.8672). We made our model and code publicly available at https://aka.ms /layoutlmv2.

Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser

“…LayoutLM (Xu et al, 2020(Xu et al, , 2021 incorporates multimodal self-supervised learning to utilize deep learning for form understanding. While it may alleviate the need for a large training dataset, it is not trivial to adopt the same method for logical structure analysis as text blocks would not fit onto the LayoutLM's context.…”

OCR-free Document Understanding Transformer

“…{ "items": [ { "name": "3002-Kyoto Choco Mochi", "count": 2, "priceInfo": { "unitPrice": 14000, "price": 28000 } }, { "name": "1001 -Choco Bun", "count": 1, "priceInfo": { "unitPrice": 22000 "price": 22000 } }, ... ], "total": [ { "menuqty_cnt": 4, "total_price": 50000 } ] } { "words": [ { "id": 1, "bbox": [[360,2048],..., [355,2127]], "text": "3002-Kyoto" }, { "id": 2, "bbox": [[801,2074],..., [801,2139]], "text": "Choco" }, { "id": 3, "bbox": [[1035,2074],..., [1035,2147]], "text": "Mochi" }, { "id": 4, "bbox": [[761,2172],..., [761,2253]], "text": "14.000" }, …, { "id": 22, "bbox": [[1573,3030],..., [1571,3126]], "text": "50.000" } ] } text information as input and perform their own objectives with the OCR-extracted texts. (Katti et al, 2018;Hwang et al, 2019Hwang et al, , 2020Hwang et al, , 2021aSage et al, 2020;Majumder et al, 2020a;Xu et al, 2019Xu et al, , 2021. For example, (Hwang et al, 2019), a currently-deployed document parsing system for business card and receipt images, consists of three separate modules for text detection, text recognition, and parsing (See Figure 2).…”

“…Earlier attempts in VDU have been done with vision-based approaches (Kang et al, 2014;Afzal et al, 2015;Harley et al, 2015a), showing the importance of textual understanding in VDU (Xu et al, 2019). With the emergence of BERT (Devlin et al, 2018), most state-of-the-arts (Xu et al, 2019(Xu et al, , 2021Hong et al, 2021) combined the computer vision (CV) and natural language processing (NLP) techniques and showed remarkable advances in recent years.…”

“…As aforementioned, current state-of-the-arts in VDU are heavily relying on large-scale real document images to train the model (Lewis et al, 2006;Xu et al, 2019Xu et al, , 2021Li et al, 2021). However, this approach is not always available in real-world production environments, in particular handling diverse languages other than English.…”

“…In some tasks, states-of-theart commercial OCR products are used. At the estimation of OCR speeds, we utilize Microsoft OCR API used in Xu et al (2021). In document parsing tasks, we use CLOVA OCR 2 specialized in the OCR on receipts and business cards images.…”