Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical, Nature, Internet and Social Media Applications

Ionescu, Bogdan; Müller, Henning; Péteri, Renaud; Abacha, Asma Ben; Sarrouti, Mourad; Demner‐Fushman, Dina; Hasan, Sadid A.; Kozlovski, Serge; Liauchuk, Vitali; Cid, Yashin Dicente; Kovalev, Vassili; Pelka, Obioma; García, Narciso; Jacutprakart, Janadhip; Friedrich, Christoph; Berari, Raul; Tauteanu, Andrei; Fichou, Dimitri; Brie, Paul; Dogariu, Mihai; Stefan, Liviu Daniel; Constantin, Mihai Gabriel; Chamberlain, Jon; Campello, Antonio; Clark, Adrian; Oliver, Thomas; Moustahfid, Hassan; Popescu, Adrian; Deshayes-Chossart, Jérôme

doi:10.1007/978-3-030-85251-1_23

Cited by 15 publications

(8 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used for our medical image captioning model evaluation, the ImageCLEFmed 2021 dataset [10], [11], which includes three sets: the training set composed of 2756 medical images; the validation set and the test set consisting of 500 and 444 radiology images, respectively. For each medical image, the medical Concepts Unique Identifiers (CUIs) and caption consisting of one or more sentences are associated.…”

Section: A Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Explainability for Medical Image Captioning

Beddiar

Oussalah

Seppänen

2022

2022 Eleventh International Conference on Image Processing Theory, Tools and Applications (IPTA)

View full text Add to dashboard Cite

Medical image captioning is the process of generating clinically significant descriptions to medical images, which has many applications among which medical report generation is the most frequent one. In general, automatic captioning of medical images is of great interest for medical experts since it offers assistance in diagnosis, disease treatment and automating the workflow of the health practitioners. Recently, many efforts have been put forward to obtain accurate descriptions but medical image captioning still provides weak and incorrect descriptions. To alleviate this issue, it is important to explain why the model produced a particular caption based on some specific features. This is performed through Artificial Intelligence Explainability (XAI), which aims to unfold the 'black-box' feature of deeplearning based models. We present in this paper an explainable module for medical image captioning that provides a sound interpretation of our attention-based encoder-decoder model by explaining the correspondence between visual features and semantic features. We exploit for that, self-attention to compute word importance of semantic features and visual attention to compute relevant regions of the image that correspond to each generated word of the caption in addition to visualization of visual features extracted at each layer of the Convolutional Neural Network (CNN) encoder. We finally evaluate our model using the ImageCLEF medical captioning dataset.

show abstract

Section: A Datasetmentioning

confidence: 99%

“…Motivated by this, we extend our proposed model [9] for the ImageCLEFmedical 2021 [10], [11] by adding an explainability module. We present therefore, an attention-based encoderdecoder model for medical image captioning.…”

Section: Introductionmentioning

confidence: 99%

Explainability for Medical Image Captioning

Beddiar

Oussalah

Seppänen

2022

2022 Eleventh International Conference on Image Processing Theory, Tools and Applications (IPTA)

View full text Add to dashboard Cite

show abstract

“…The dataset is provided by ImageCLEF Tuberculosis Type 2021 Challenge [11,12]. It contains CT scans of a total of 1338 TB patients, 917 of them have been used for training and 421 for the test set.…”

Section: Datasets and Preprocessingmentioning

confidence: 99%

ViPTT-Net: Video pretraining of spatio-temporal model for tuberculosis type classification from chest CT scans

Zunair,

Rahman,

Mohammed

2021

Preprint

View full text Add to dashboard Cite

Pretraining has sparked groundswell of interest in deep learning workflows to learn from limited data and improve generalization. While this is common for 2D image classification tasks, its application to 3D medical imaging tasks like chest CT interpretation is limited. We explore the idea of whether pretraining a model on realistic videos could improve performance rather than training the model from scratch, intended for tuberculosis type classification from chest CT scans. To incorporate both spatial and temporal features, we develop a hybrid convolutional neural network (CNN) and recurrent neural network (RNN) model, where the features are extracted from each axial slice of the CT scan by a CNN, these sequence of image features are input to a RNN for classification of the CT scan. Our model termed as ViPTT-Net, was trained on over 1300 video clips with labels of human activities, and then fine-tuned on chest CT scans with labels of tuberculosis type. We find that pretraining the model on videos lead to better representations and significantly improved model validation performance from a kappa score of 0.17 to 0.35, especially for under-represented class samples. Our best method achieved 2nd place in the ImageCLEF 2021 Tuberculosis -TBT classification task with a kappa score of 0.20 on the final test set with only image information (without using clinical meta-data). All codes and models are made available. 1

show abstract

“…To solve this problem, Sarrouti et al [ 21 ] proposed a variational auto-encoder (VAE) [ 22 ] model to generate questions from radiology images. Many methods have been proposed in the “imageCLEF” [ 23 , 24 ], a medical image processing competition. Since 2020, medical VQG has been included in the competition, and many contestants have proposed methods for improving the accuracy of medical VQG.…”

Section: Introductionmentioning

confidence: 99%

Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation

Zhu

Togo

Ogawa

et al. 2023

Sensors

View full text Add to dashboard Cite

Auxiliary clinical diagnosis has been researched to solve unevenly and insufficiently distributed clinical resources. However, auxiliary diagnosis is still dominated by human physicians, and how to make intelligent systems more involved in the diagnosis process is gradually becoming a concern. An interactive automated clinical diagnosis with a question-answering system and a question generation system can capture a patient’s conditions from multiple perspectives with less physician involvement by asking different questions to drive and guide the diagnosis. This clinical diagnosis process requires diverse information to evaluate a patient from different perspectives to obtain an accurate diagnosis. Recently proposed medical question generation systems have not considered diversity. Thus, we propose a diversity learning-based visual question generation model using a multi-latent space to generate informative question sets from medical images. The proposed method generates various questions by embedding visual and language information in different latent spaces, whose diversity is trained by our newly proposed loss. We have also added control over the categories of generated questions, making the generated questions directional. Furthermore, we use a new metric named similarity to accurately evaluate the proposed model’s performance. The experimental results on the Slake and VQA-RAD datasets demonstrate that the proposed method can generate questions with diverse information. Our model works with an answering model for interactive automated clinical diagnosis and generates datasets to replace the process of annotation that incurs huge labor costs.

show abstract

Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical, Nature, Internet and Social Media Applications

Cited by 15 publications

References 20 publications

Explainability for Medical Image Captioning

Explainability for Medical Image Captioning

ViPTT-Net: Video pretraining of spatio-temporal model for tuberculosis type classification from chest CT scans

Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation

Contact Info

Product

Resources

About