A comprehensive interpretation for medical VQA: Datasets, techniques, and challenges

Abstract

Recent technological developments and improvement in the medical domain demands advancement, to address the issue of early disease detection. Also, the current pandemic has resulted in considerable progress of improvement in the medical domain, through online consultation by physicians for different diseases using clinical reports and medical images. A similar process is adopted in developing a Visual Question Answering (VQA) system in the medical field. In this paper, existing medical VQA datasets, appropriate techniques, suitable quantitative metrics, real time challenges and, the implementation of one VQA approach with algorithms and performance evaluation are discussed. The medical VQA datasets collected from multiple sources are represented in different perspectives (organwise, planewise, modality-type and abnormality-type) for a better understanding and visualization. Then the techniques used in VQA are subsequently grouped and explained, based on evolution, complexity in the dataset and the need for semantics in understanding the questions. In addition, the implementation of a VQA approach using VGGNet and LSTM is carried out for existing and improved datasets, and analyzed with accuracy and BLEU score metrics. The improved datasets, created through dataset reduction and augmentation approaches, resulted in better performance than the existing datasets. Finally, the challenges of the medical VQA domain are examined in terms of datasets, combining techniques, and modifying the parameters of existing performance metrics for future research.

A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering

“…Table 1 shows the published survey research in the VQA field from 2017 to 2023. Although many surveys were published, which denoted how active this field is, most of these publications were in the general field and did not have sections on medical VQA, their pros and cons, or the open challenges in the medical field except Lin et al [24] (2021), Noor Mohammed and Srinivasan [26] (2023), and Lin et al [27] (2023). Lin et al [24] and Noor Mohamed and Srinivasan [26] propose a VQA survey study in the medical field.…”

“…Although many surveys were published, which denoted how active this field is, most of these publications were in the general field and did not have sections on medical VQA, their pros and cons, or the open challenges in the medical field except Lin et al [24] (2021), Noor Mohammed and Srinivasan [26] (2023), and Lin et al [27] (2023). Lin et al [24] and Noor Mohamed and Srinivasan [26] propose a VQA survey study in the medical field. Those survey studies survey the methods designed for the 2018, 2019, and 2020 ImageCLEF challenges and two extra models on the VQA-RAD dataset.…”

“…Those survey studies survey the methods designed for the 2018, 2019, and 2020 ImageCLEF challenges and two extra models on the VQA-RAD dataset. The Lin et al [24] study also surveys the public medical VQA datasets except VQA-Med 2021, whereas Noor Mohammed and Srinivasan [26] added VQA-Med 2021 and Diabetic Macular Edema (DME) dataset [28]. Additionally, they discussed the challenges and future studies in the field.…”

“…Additionally, they discussed the challenges and future studies in the field. Lin et al [24] and Noor Mohamed and Srinivasan [26] surveys, which are based on a comparison of 32 studies, require further work to cover and analyze more methods and datasets. The most recent survey, Lin et al [27], is more comprehensive than the previous two surveys [24], [26] where it covers 44 studies with 47 models.…”

“…Lin et al [24] and Noor Mohamed and Srinivasan [26] surveys, which are based on a comparison of 32 studies, require further work to cover and analyze more methods and datasets. The most recent survey, Lin et al [27], is more comprehensive than the previous two surveys [24], [26] where it covers 44 studies with 47 models. The present survey study is an analytical review with 60 studies in medical VQA.…”