Document Collection Visual Question Answering

Abstract

Current tasks and methods in Document Understanding aims to process documents as single elements. However, documents are usually organized in collections (historical records, purchase invoices), that provide context useful for their interpretation. To address this problem, we introduce Document Collection Visual Question Answering (DocCVQA) a new dataset and related task, where questions are posed over a whole collection of document images and the goal is not only to provide the answer to the given question, but also to retrieve the set of documents that contain the information needed to infer the answer. Along with the dataset we propose a new evaluation metric and baselines which provide further insights to the new dataset and task.

OCR-IDL: OCR Annotations for Industry Document Library Dataset

“…Document Intelligence can be considered as an umbrella term covering problems of Key Information Extraction [10,54], Table Detection [41,38] and Structure Recognition [39,55], Document Layout Segmentation [5,4] Document Layout Generation [6,36,3,48], Document Visual Question Answering [51,50,32], Document Image Enhancement [49,22,47] which involves the understanding of visually rich semantic information and structure of different layout entities of a whole page.…”

“…Secondly, we decide to use a commercial OCR engine, specifically Amazon Textract 3 , over Tesseract. It is because the performance of the OCR engines can significantly affect the model's performance which can be seen in fields that use OCR annotations, such as in fine-grained classification [29,30,31], in scenetext visual question answering [9,44,8,13], in document visual question answering (DocVQA) [50,33]. Apart from improving the annotation quality significantly, we want to level the differences between research groups and companies.…”

“…IDL is a digital archive of documents created by industries which influence public health, hosted by the University of California, San Francisco Library 4 . IDL has already been used in the literature for building datasets: IIT-CDIP [25], RVL-CDIP [18], DocVQA [50,33]. Hence, our OCR annotations can be used to further advance in these tasks.…”