ScanBank: A Benchmark Dataset for Figure Extraction from Scanned Electronic Theses and Dissertations

Figure and Figure Caption Extraction for Mixed Raster and Vector PDFs: Digitization of Astronomical Literature with OCR Features

“…Through ablation experiments we find the combination of the page and hOCR properties of (grayscale, ascenders, decenders, word confidences, fraction of numbers in a word, fraction of letters in a word, punctuation, word rotation and spaCy POS) maximize our model's performance. When compared to other deep learning models popular for document layout analysis (ScanBank [21,47] and de-tectron2 [45]) we find our model performs better on our dataset, particularly at the high IOU thresholds (IOU=0.9) and especially for figure captions. In particular, in line with our extraction goals, our model has relatively low false positive rates, minimizing the extraction of erroneous page objects.…”

“…Using the definitions of captions from Table 3 for 201 captions the results are more promising with F1 scores of 65.6% (ours) to 65.5%(detectron2). Results for our (ScanBank's) model applied to the ScanBank collection of "gold-standard" ETDs [21,47] are lower overall with F1 scores of 25.5%(38.4%) for 197 figures and 16.3%(1.4%) for 140 captions. While these results suggest that our model may be more generalizable than other models for figure captions, tests on larger datasets are necessary for a firmer conclusion.…”

“…Table 3 shows how other deep learning models fair on our final test dataset. Here, we use ScanBank [21,47] (based on DeepFigures [38] and trained on a corpus of pre-digital electronic thesis and dissertations (ETDs)) and a version of detectron2 [45] trained on the PubLayNet dataset [52]. ScanBank and detectron2 are used for comparison as they are applied to raster-formatted articles (as opposed to vector-based methods like pdffigures2 [11] which, applied to our data, results in F1 scores of <15% in feature and final test splits).…”

“…As discussed in [21], ScanBank's figure's are defined as encompassing the figure caption, while our figure definitions exclude the caption. In order to compare with our results, we initially performed metric calculations by re-defining our true figure boxes as the combination of figure and figure caption boxes when figure captions are associated with a figure.…”

“…Table 3. Performance metrics for ScanBank [21,47] and detectron2 [45] for our final test dataset. IOU is 0.9.…”