AMAM: An Attention-based Multimodal Alignment Model for Medical Visual Question Answering

Pan, Haiwei; He, Shuning; Zhang, Kejia; Chen, Chunling; Shi, K.

doi:10.1016/j.knosys.2022.109763

Cited by 15 publications

(1 citation statement)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Scholars from various countries have proposed many VQA models using deep neural networks and achieved good prediction results. At present, these models can be divided into four categories, namely, models of attention-based [6], Bilinear Superdiagonal Fusion [7], Multimodal Relational Reasoning [8] and external knowledge-based [9]. Ren et al used VGG19 algorithm to extract features of images, and then input natural language questions into LSTM model as vectors.…”

Section: Related Workmentioning

confidence: 99%

Research on visual question answering system based on deep neural network model

Tang

2023

Fifth International Conference on Computer Information Science and Artificial Intelligence (CISAI 2022)

View full text Add to dashboard Cite

With the rapid development of deep learning, computer vision, natural semantic processing and other technologies, Visual Question Answering (VQA) has gradually become to be an important research direction in the multimodal field. VQA technology has a wide range of application scenarios, such as multi-modal search, medical consultation, intelligent driving and so on, and has become the research hotspot and focus of scholars. Based on the classical CLEVR dataset, this paper builds different VQA prediction models by using the BilSTM model and different deep learning algorithms, including modnet, VGG and ResNet models. The results show that the hybrid model of ResNet and BiLSTM has the highest prediction accuracy, reaching 0.978. Compared with other algorithms, the accuracy is 1.3%, 2.5% and 2.7% higher than that of MobileMet, VGG19 and VGG16 models, which means that the hybrid model of resnet50 and BiLSTM is more effective in VQA tasks. At the same time, some problems and deficiencies of our model are analyzed and the future improvement direction is given.

show abstract