MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering

Abstract

Medical V isual Question Answering (VQA) is a multi-modal challenging task widely considered by research communities of the computer vision and natural language processing. Since most current medical VQA models focus on visual content, ignoring the importance of text, this paper proposes a multi-v iew attentionbased model(MuVAM) for medical visual question answering which integrates the high-level semantics of medical images on the basis of text description. Firstly, different methods are utilized to extract the features of the image and the question for the two modalities of vision and text. Secondly, this paper proposes a multi-view attention mechanism that include Image-to-Question (I2Q) attention and W ord-to-T ext (W2T) attention. Multi-view attention can correlate the question with image and word in order to better analyze the question and get an accurate answer. Thirdly, a composite loss is presented to predict the answer accurately after multi-modal feature fusion and improve the similarity between visual and textual cross-modal features. It consists of classification loss and image-question complementary (IQC) loss. Finally, for data errors and missing labels in the VQA-RAD dataset, we collaborate with medical experts to correct and complete this dataset and then construct an enhanced dataset, VQA-RAD Ph . The experiments on these two datasets show that the effectiveness of MuVAM surpasses the state-of-the-art method.

Is Unimodal Bias Always Bad for Visual Question Answering? A Medical Domain Study with Dynamic Attention

“…In practice, doctors are required to have a profound understanding of the problems indicated by medical images and perform explicit reasoning to confirm a diagnosis [4], the Fig. 1.…”

“…Learning tasks involved in VQA, cited from [3] process of which may be lengthy and costly. Instead, the medical visual question answering task can better assist the doctor in the diagnosis and alleviate the imbalanced medical resource status [5], by significantly reducing misdiagnosis and improving accuracy [4].…”

“…Attention mechanisms, including Bilinear Attention Networks (BAN) [9] and Stacked Attention Networks (SAN) [10] are also deployed, to enhance the relation caption of visual and textual information. Recent advances include reasoning [1], to consider closed -and open-ended questions, multi-view attention [4], to correlate questions with images and texts with more attention, and Multiple Meta-model Quantifying (MMQ) [11], to enhance meta-data, by auto-annotation with noisy labels.…”

“…It assists models in understanding the relations between questions and images deeply and is necessary when the questions are complex or difficult to answer. The use of attention can significantly improve the performance of models on VQA tasks [4]. [1] proposed a question-conditioned reasoning model, which selectively fused the multimodal features, according to their importance, and learned more semantic information from the question representations.…”

“…[1] proposed a question-conditioned reasoning model, which selectively fused the multimodal features, according to their importance, and learned more semantic information from the question representations. [4] then designed the multi-view attention model, including image-to-question and word-to-text attention, to correlate questions with images. In this paper, we apply a different expressive attention method, known as dynamic graph attention, introduced by [18], which we show to outperform the state-of-the-art.…”