DocLayNet: A Large Human-Annotated Dataset for Document-Layout Segmentation

Pfitzmann, Birgit; Auer, Christoph; Dolfi, Michele; Nassar, Ahmed; Staar, Peter

doi:10.1145/3534678.3539043

Cited by 38 publications

(28 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The lack of generalizability is a known and pervasive problem in the field of document layout analysis [e.g. 6,13]. A change in not only publication type, but simply publication year can drastically lower the accuracy of page object extraction methods for models that are not explicitly trained on this type of document [10,12,14].…”

Section: The Problem Of Generalizabilitymentioning

confidence: 99%

“…are looking for on a document's page. As discussed in [14], When different annotators work on the same dataset, they can disagree on the class definitions leading to inconsistent data within the same dataset [13,20]. While there have been pushes to adopt a standardized methodology for defining page object classes [e.g.…”

Section: Inconsistent Page Object Definitionsmentioning

confidence: 99%

“…Assuming there is agreement within a group or field as to the definitions of page object classes, one option for generating data is to employ a large group of annotators to hand-classify article pages. One such effort is the DocLayNet dataset [13]. This large scale annotation effort illustrates many of the logistic and resource requirements for the generation of a large and diverse set of documents for use with machine learning models.…”

Section: Expensive Human Annotationmentioning

confidence: 99%

“…The DocLayNet dataset, which contains over 80,000 manually annotated pages, took ∼40 supervised annotators and a small group of experts over 6 months to produce, with only a "small fraction" of the pages being seen by more than one annotator [13]. Additionally, while the annotation process included a >100 page annotation guide and a 12 week training period, the resulting intercoder agreement was ≈80-85% for their 11 page object categories in the "small fraction" of double and triple-annotated pages [13].…”

Section: Expensive Human Annotationmentioning

confidence: 99%

“…The significant logistics and time requirements for the generation of this dataset, and its resulting intercoder agreement of 80-85% for each class demonstrates how difficult these hand-annotation tasks are to scale. Relevant to our work, this dataset does not include scanned pages -the DocLayNet authors point out these pages are often warped or non-uniformly colored introducing even more possible uncertainty and errors into the annotation process [13].…”

Section: Expensive Human Annotationmentioning

confidence: 99%

See 4 more Smart Citations

Generalizability in Document Layout Analysis for Scientific Article Figure & Caption Extraction

Naiman¹

2023

Preprint

View full text Add to dashboard Cite

The lack of generalizability -in which a model trained on one dataset cannot provide accurate results for a different dataset -is a known problem in the field of document layout analysis. Thus, when a model is used to locate important page objects in scientific literature such as figures, tables, captions, and math formulas, the model often cannot be applied successfully to new domains. While several solutions have been proposed, including newer and updated deep learning models, larger handannotated datasets, and the generation of large synthetic datasets, so far there is no "magic bullet" for translating a model trained on a particular domain or historical time period to a new field. Here we present our ongoing work in translating our document layout analysis model from the historical astrophysical literature to the larger corpus of scientific documents within the HathiTrust U.S. Federal Documents collection. We use this example as an avenue to highlight some of the problems with generalizability in the document layout analysis community and discuss several challenges and possible solutions to address these issues. All code for this work is available on The Reading Time Machine GitHub repository, https://github.com/ReadingTimeMachine/htrc short conf.

show abstract