Deep Multimodal Classification of Image Types in Biomedical Journal Figures

Andrearczyk, Vincent; Müller, Henning

doi:10.1007/978-3-319-98932-7_1

Cited by 15 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To automatically assign class label to individual panels, we build an image classifier. A pre-trained Convolutional Neural Network, VGG16 (Andrearczyk and M ü ller, 2018 ; Simonyan and Zisserman, 2015 ), is used for image classification. To train the classifier, we use the annotated image dataset that was introduced by Lopez et al (2013) based on the Molecular INTeraction database dataset ( Licata et al , 2012 ).…”

Section: Methodsmentioning

confidence: 99%

Utilizing image and caption information for biomedical document classification

Jiang

Zhang

et al. 2021

Bioinformatics

View full text Add to dashboard Cite

Motivation Biomedical research findings are typically disseminated through publications. To simplify access to domain-specific knowledge while supporting the research community, several biomedical databases devote significant effort to manual curation of the literature—a labor intensive process. The first step toward biocuration requires identifying articles relevant to the specific area on which the database focuses. Thus, automatically identifying publications relevant to a specific topic within a large volume of publications is an important task toward expediting the biocuration process and, in turn, biomedical research. Current methods focus on textual contents, typically extracted from the title-and-abstract. Notably, images and captions are often used in publications to convey pivotal evidence about processes, experiments and results. Results We present a new document classification scheme, using both image and caption information, in addition to titles-and-abstracts. To use the image information, we introduce a new image representation, namely Figure-word, based on class labels of subfigures. We use word embeddings for representing captions and titles-and-abstracts. To utilize all three types of information, we introduce two information integration methods. The first combines Figure-words and textual features obtained from captions and titles-and-abstracts into a single larger vector for document representation; the second employs a meta-classification scheme. Our experiments and results demonstrate the usefulness of the newly proposed Figure-words for representing images. Moreover, the results showcase the value of Figure-words, captions and titles-and-abstracts in providing complementary information for document classification; these three sources of information when combined, lead to an overall improved classification performance. Availability and implementation Source code and the list of PMIDs of the publications in our datasets are available upon request.

show abstract

Section: Methodsmentioning

confidence: 99%

Utilizing image and caption information for biomedical document classification

Jiang

Zhang

et al. 2021

Bioinformatics

View full text Add to dashboard Cite

show abstract

“…Several challenges on identifying the image types were run in ImageCLEF [15]. The currently best results for the really challenging and very unbalanced data set of over 30 classes reached over 90% [2].…”

Section: Extracting Content From Medical Imagesmentioning

confidence: 99%

“…In PMC, most articles have manually attached MeSH (Medical Subject Headings) terms. Text has been used in most retrieval applications [10] but has also obtained very good results in modality classification [17,2], as it is complementary to visual information. In the case of compound figure detection, a caption with several subparts can also be indicative for the presence of subfigures.…”

Section: Combining Text and Images For Data Analysismentioning

confidence: 99%

Studying Public Medical Images from the Open Access Literature and Social Networks for Model Training and Knowledge Extraction

Müller

Andrearczyk

Toro

et al. 2019

MultiMedia Modeling

Self Cite

View full text Add to dashboard Cite

Medical imaging research has long suffered problems getting access to large collections of images due to privacy constraints and to high costs that annotating images by physicians causes. With public scientific challenges and funding agencies fostering data sharing, repositories, particularly on cancer research in the US, are becoming available. Still, data and annotations are most often available on narrow domains and specific tasks. The medical literature (particularly articles contained in MedLine) has been used for research for many years as it contains a large amount of medical knowledge. Most analyses have focused on text, for example creating semi-automated systematic reviews, aggregating content on specific genes and their functions, or allowing for information retrieval to access specific content. The amount of research on images from the medical literature has been more limited, as MedLine abstracts are available publicly but no images are included. With PubMed Central, all the biomedical open access literature has become accessible for analysis, with images and text in structured format. This makes the use of such data easier than extracting it from PDF. This article reviews existing work on analyzing images from the biomedical literature and develops ideas on how such images can become useful and usable for a variety of tasks, including finding visual evidence for rare or unusual cases. These resources offer possibilities to train machine learning tools, increasing the diversity of available data and thus possibly the robustness of the classifiers. Examples with histopathology data available on Twitter already show promising possibilities. This article add links to other sources that are accessible, for example via the ImageCLEF challenges.

show abstract

“…This data set was changed strongly compared to the same task run in 2017 to reduce the diversity on the data and limit the number of compound figures. A subset of clinical figures was automatically obtained from the overall set of 5.8 million PMC figures using a deep multimodal fusion of Convolutional Neural Networks (CNN), described in [2]. In total, the dataset is comprised of 232,305 imagecaption pairs split into disjoint training (222,305 pairs) and test (10,000 pairs) sets.…”

Section: Datasetmentioning

confidence: 99%

Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation

Ionescu

Müller

Villegas³

et al. 2018

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This paper presents an overview of the ImageCLEF 2018 evaluation campaign, an event that was organized as part of the CLEF (Conference and Labs of the Evaluation Forum) Labs 2018. ImageCLEF is an ongoing initiative (it started in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval with the aim of providing information access to collections of images in various usage scenarios and domains. In 2018, the 16th edition of ImageCLEF ran three main tasks and a pilot task: (1) a caption prediction task that aims at predicting the caption of a figure from the biomedical literature based only on the figure image; (2) a tuberculosis task that aims at detecting the tuberculosis type, severity and drug resistance from CT (Computed Tomography) volumes of the lung; (3) a LifeLog task (videos, images and other sources) about daily activities understanding and moment retrieval, and (4) a pilot task on visual question answering where systems are tasked with answering medical questions. The strong participation, with over 100 research groups registering and 31 submitting results for the tasks, shows an increasing interest in this benchmarking campaign.

show abstract

Deep Multimodal Classification of Image Types in Biomedical Journal Figures

Cited by 15 publications

References 13 publications

Utilizing image and caption information for biomedical document classification

Utilizing image and caption information for biomedical document classification

Studying Public Medical Images from the Open Access Literature and Social Networks for Model Training and Knowledge Extraction

Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation

Contact Info

Product

Resources

About