CorDeep and the Sacrobosco Dataset: Detection of Visual Elements in Historical Documents

Abstract

Recent advances in object detection facilitated by deep learning have led to numerous solutions in a myriad of fields ranging from medical diagnosis to autonomous driving. However, historical research is yet to reap the benefits of such advances. This is generally due to the low number of large, coherent, and annotated datasets of historical documents, as well as the overwhelming focus on Optical Character Recognition to support the analysis of historical documents. In this paper, we highlight the importance of visual elements, in particular illustrations in historical documents, and offer a public multi-class historical visual element dataset based on the Sphaera corpus. Additionally, we train an image extraction model based on YOLO architecture and publish it through a publicly available web-service to detect and extract multi-class images from historical documents in an effort to bridge the gap between traditional and computational approaches in historical studies.

Explainability and transparency in the realm of digital humanities: toward a historian XAI

“…Electronic copies of these books are available via the project's database, comprising over 70,000 pages, 23,000 of which contain visual elements. These visual elements were collected both manually and with the help of neural networks (Büttner et al, 2022). The Sphaera dataset is stored in a large knowledge graph modeled according to the CIDOC-CRM standards (Bekiari et al, 2021), where information about the editions, as well as fine-grained information about their content is stored (Kräutli & Valleriani, 2018;El-Hajj et al, 2022).…”

“…The second added class refers to illustrations of material objects, namely those which one can refer to as machines. This data was collected from Branca (1629); Zonca (1607); Ramelli (1588); Besson (1595) using CorDeep (https://cordeep.mpiwg-berlin.mpg.de), a web service designed to extract and classify visual elements from historical documents (Büttner et al, 2022). In total, the dataset contains 5,879 pages distributed across the four classes, as shown in Table 1; each class is represented by a single sample in Fig.…”

“…These are selected by domain experts and labeled accordingly. To facilitate this process, illustrations are extracted from full book pages using the automated image segmentation pipeline CorDeep (Büttner et al, 2022). A web service to extract visual elements from various input types including PDF and common image file formats is accessible via https://cordeep.mpiwg-berlin.…”

“…In historical document layout analysis in particular, e.g., in Xu et al (2018), the authors relied on a Multi-Task Fully Convolutional Network (FCN) to segment highly unstructured manuscript and printed-text pages into multiple semantically relevant groups (e.g., marginalia, main text, and comments), while Ravichandra et al (2022) opts for an object-detection based approach relying on the YOLO model (Redmon et al, 2015). Others have recognized the value of extracting images from historical documents due to their importance in transmitting the information and ideas contained in the texts, leading to approaches such as the FCN networks presented in Monnier and Aubry (2020) and the object detection-based methodologies applied to specific corpora adopted by Dutta et al (2021); Büttner et al (2022) from techniques like YOLO (Redmon et al, 2015), U-Net (Ronneberger et al, 2015), or Faster R-CNN (Ren et al, 2016). By getting closer to the textual content of these documents, numerous AI-based approaches for optical character recognition (OCR) and handwritten text recognition (HTR) have been proposed, with deep learning-based approaches (Jaderberg et al, 2016) setting new standards.…”

Line-Level Layout Recognition of Historical Documents with Background Knowledge

“…Afterwards, the results were injected into Kraken [23] to generate text lines and the OCR output. Likewise, Büttner et al [24] applied a YoloV5 network to detect graphical elements, such as initials, decorations, printer's marks, or content illustrations, in different historical documents.…”