Deconfounded and Explainable Interactive Vision-Language Retrieval of Complex Scenes

Abstract

In vision-language retrieval systems, users provide natural language feedback to find target images. Vision-language explanations in the systems can better guide users to provide feedback and thus improve the retrieval. However, developing explainable vision-language retrieval systems can be challenging, due to limited labeled multimodal data. In the retrieval of complex scenes, the issue of limited labeled data can be more severe. With multiple objects in the complex scenes, each user query may not exhaustively describe all objects in the desired image and thus more labeled queries are needed. The issue of limited labeled data can cause data selection biases, and result in spurious correlations learned by the models. When learning spurious correlations, existing explainable models may not be able to accurately extract regions from images and keywords from user queries.In this paper, we discover that deconfounded learning is an important step to provide better vision-language explanations. Thus we propose a deconfounded explainable vision-language retrieval system. By introducing deconfounded learning to pretrain our vision-language model, the spurious correlations in the model can be reduced through interventions by potential confounders. This helps to train more accurate representations and further enable better explainability. Based on explainable retrieval results, we propose novel interactive mechanisms. In such interactions, users can better understand why the system returns particular results and give feedback effectively improving the results. This additional feedback is sample efficient and thus alleviates the data limitation problem. Through extensive experiments, our system achieves about 60% improvements, compared to the state-of-the-art. CCS CONCEPTS• Computing methodologies → Causal reasoning and diagnostics; • Information systems → Users and interactive retrieval; Retrieval efficiency; Presentation of retrieval results; Image search.

A Survey on Trustworthy Recommender Systems

“…The term "Explainable Recommendation" was first defined by Zhang et al [341]. As an important sub-field of AI and machine learning research and due to the fact that recommendation naturally involves humans in the loop, the recommender system community has been leading the research on Explainable AI ever since, which triggers a broader scope of explainability research in other AI and machine learning sub-fields [71,340], such as explainability in scientific research [181], computer vision [297], natural language processing [40,106,172,217,229], graph neural networks [265,299], database [112,291], healthcare systems [121,228,350], online education [9,20,216,264,277], psychological studies [271] and cyber-physical systems [10,12,134,135,241].…”

“…Omidvar-Tehrani et al [215] conduct Explainable Points-of-Interest (POI) Recommendation as an exploratory process in which users are allowed to keep interaction with the system explanations by expressing their favorite POIs, and the interactions will impact the recommendation process. Wu et al [297] provide vision-language explanations by deconfounded learning that conducts pre-training for the vision-language model. In that way, the potential effects of confounders are removed, which will expedite accurate representation training and better explainability.…”