Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation

Abstract

Auxiliary clinical diagnosis has been researched to solve unevenly and insufficiently distributed clinical resources. However, auxiliary diagnosis is still dominated by human physicians, and how to make intelligent systems more involved in the diagnosis process is gradually becoming a concern. An interactive automated clinical diagnosis with a question-answering system and a question generation system can capture a patient’s conditions from multiple perspectives with less physician involvement by asking different questions to drive and guide the diagnosis. This clinical diagnosis process requires diverse information to evaluate a patient from different perspectives to obtain an accurate diagnosis. Recently proposed medical question generation systems have not considered diversity. Thus, we propose a diversity learning-based visual question generation model using a multi-latent space to generate informative question sets from medical images. The proposed method generates various questions by embedding visual and language information in different latent spaces, whose diversity is trained by our newly proposed loss. We have also added control over the categories of generated questions, making the generated questions directional. Furthermore, we use a new metric named similarity to accurately evaluate the proposed model’s performance. The experimental results on the Slake and VQA-RAD datasets demonstrate that the proposed method can generate questions with diverse information. Our model works with an answering model for interactive automated clinical diagnosis and generates datasets to replace the process of annotation that incurs huge labor costs.

Multimodal Natural Language Explanation Generation for Visual Question Answering Based on Multiple Reference Data

“…Deep learning has advanced rapidly in recent years, resulting in the practical application of artificial intelligence systems in various scenarios, and having a significant impact on society [1]. Since a multi-step reasoning task involves computer vision (CV) [2] and natural language processing (NLP) [3], visual question answering (VQA) [4] has become one of the most important areas of development for detecting defects in flow lines [5] and for complementary diagnostics [6,7]. Because these applications are directly related to the safety of people's lives and property, new regulations have been introduced, requiring model reliability and increasing the demand for model interpretability [8,9].…”