Image retrieval from scientific publications: Text and image content processing to separate multipanel figures

Abstract

Images contained in scientific publications are widely considered useful for educational and research purposes, and their accurate indexing is critical for efficient and effective retrieval. Such image retrieval is complicated by the fact that figures in the scientific literature often combine multiple individual subfigures (panels). Multipanel figures are in fact the predominant pattern in certain types of scientific publications. The goal of this work is to automatically segment multipanel figures-a necessary step for automatic semantic indexing and in the development of image retrieval systems targeting the scientific literature. We have developed a method that uses the image content as well as the associated figure caption to: (1) automatically detect panel boundaries; (2) detect panel labels in the images and convert them to text; and (3) detect the labels and textual descriptions of each panel within the captions. Our approach combines the output of imagecontent and text-based processing steps to split the multipanel figures into individual subfigures and assign to each subfigure its corresponding section of the caption. The developed system achieved precision of 81% and recall of 73% on the task of automatic segmentation of multipanel figures.

Evaluating performance of biomedical image retrieval systems—An overview of the medical image retrieval task at ImageCLEF 2004–2013

“…On the other hand, only one group applied textual techniques (in combination with visual techniques) for the compound figure separation task. This group determined the number of image panels comprising a compound figure by identifying textual panel labels in the figure’s caption [1]. …”

Separating compound figures in journal articles to allow for subfigure classification

“…Cheng, Antani et al 12 used two methods based on the modality to improve the algorithm but they tested their implementation on a different dataset. Apostolova et al 17 have presented their results on a subset of the open ImageCLEF benchmark but the actual dataset has not been made available publicly.…”

Stitched Multipanel Biomedical Figure Separation

“…In this context, multipanel figure separation is considered as a crucial step, assuming that each subpanel contains a single modality. By doing so, it improves the performance of CBIR [4], [16]- [18], and hence is a precursor to biomedical CBIR.…”

“…Projection profile-based methods are commonly used (and fairly sufficient) to separate subpanels of those figures, which are having homogeneous gaps between them [17], [19]. Either of the two penetrations (using horizontal and vertical profiles) helps to separate them (i.e., two different levels of separations are required [17], [18]). But, when homogeneous gaps do not penetrate from left to right and top to bottom, straightforward projection profiles fail to separate subpanels.…”