Utilizing image and caption information for biomedical document classification

Abstract

Motivation Biomedical research findings are typically disseminated through publications. To simplify access to domain-specific knowledge while supporting the research community, several biomedical databases devote significant effort to manual curation of the literature—a labor intensive process. The first step toward biocuration requires identifying articles relevant to the specific area on which the database focuses. Thus, automatically identifying publications relevant to a specific topic within a large volume of publications is an important task toward expediting the biocuration process and, in turn, biomedical research. Current methods focus on textual contents, typically extracted from the title-and-abstract. Notably, images and captions are often used in publications to convey pivotal evidence about processes, experiments and results. Results We present a new document classification scheme, using both image and caption information, in addition to titles-and-abstracts. To use the image information, we introduce a new image representation, namely Figure-word, based on class labels of subfigures. We use word embeddings for representing captions and titles-and-abstracts. To utilize all three types of information, we introduce two information integration methods. The first combines Figure-words and textual features obtained from captions and titles-and-abstracts into a single larger vector for document representation; the second employs a meta-classification scheme. Our experiments and results demonstrate the usefulness of the newly proposed Figure-words for representing images. Moreover, the results showcase the value of Figure-words, captions and titles-and-abstracts in providing complementary information for document classification; these three sources of information when combined, lead to an overall improved classification performance. Availability and implementation Source code and the list of PMIDs of the publications in our datasets are available upon request.

Simplifying Multimodal Emotion Recognition with Single Eye Movement Modality

“…Text usually teams up with image or audio to boost the performance of models in many tasks. For example, Jain et al [19], Audebert et al [2], and Li et al [25] used semantic information together with visual cues to classify documents. Besides, many studies [32,14] in captioning and image classification also rely on the cooperation between caption/description and image.…”

Hagit Shatkay-Reshef 1965–2022

“…I was very lucky to work with her and Drs Kambhamettu (University of Delaware) and Marai (University of Illinois Chicago), as well as colleagues from MGI and Wormbase databases, in a project aimed at incorporating image-based features into biomedical document classification. Not only was this a very fruitful research experience ( Jiang et al , 2019 ; Li et al , 2021 ; Trabucco et al , 2021 ), but also a great opportunity to work with an outstanding researcher, who was very rigorous, and had strong work ethics. This sentiment was echoed by PI Dr Marai, ‘I met Hagit when I was a graduate student, and we became friends.…”