Explainability for Medical Image Captioning

Abstract

Medical image captioning is the process of generating clinically significant descriptions to medical images, which has many applications among which medical report generation is the most frequent one. In general, automatic captioning of medical images is of great interest for medical experts since it offers assistance in diagnosis, disease treatment and automating the workflow of the health practitioners. Recently, many efforts have been put forward to obtain accurate descriptions but medical image captioning still provides weak and incorrect descriptions. To alleviate this issue, it is important to explain why the model produced a particular caption based on some specific features. This is performed through Artificial Intelligence Explainability (XAI), which aims to unfold the 'black-box' feature of deeplearning based models. We present in this paper an explainable module for medical image captioning that provides a sound interpretation of our attention-based encoder-decoder model by explaining the correspondence between visual features and semantic features. We exploit for that, self-attention to compute word importance of semantic features and visual attention to compute relevant regions of the image that correspond to each generated word of the caption in addition to visualization of visual features extracted at each layer of the Convolutional Neural Network (CNN) encoder. We finally evaluate our model using the ImageCLEF medical captioning dataset.

DICKT—Deep Learning-Based Image Captioning using Keras and TensorFlow

“…It makes use of strategies such as Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNN) to understand and interpret images, allowing the machine to offer captions in a commonplace language [7]. This generation has various applications, along with supporting blind individuals by way of presenting braille legible captions for photos, aiding in medical document generation, and automating the workflow of healthcare experts [8], [9]. However, the heavy computational burden and massive memory storage required by way of present captioning fashions restrict their deployment on aid-restricted gadgets.…”

Interpretability and Transparency of Machine Learning in File Fragment Analysis with Explainable Artificial Intelligence

“…It focuses on developing techniques and methodologies that enable humans to understand and interpret the decisions made by machine learning models. Explainable artificial intelligence aims to provide meaningful explanations for the outcomes produced by AI models, enhancing transparency and understanding of their reasoning [6]. Explainable AI makes artificial intelligence models more manageable and understandable.…”