Interactive Re-ranking via Object Entropy-Guided Question Answering for Cross-Modal Image Retrieval

Abstract

Cross-modal image-retrieval methods retrieve desired images from a query text by learning relationships between texts and images. Such a retrieval approach is one of the most effective ways of achieving the easiness of query preparation. Recent cross-modal image-retrieval methods are convenient and accurate when users input a query text that can be used to uniquely identify the desired image. However, in reality, users frequently input ambiguous query texts, and these ambiguous queries make it difficult to obtain desired images. To overcome these difficulties, in this study, we propose a novel interactive cross-modal image-retrieval method based on question answering. The proposed method analyzes candidate images and asks users questions to obtain information that can narrow down retrieval candidates. By only answering questions generated by the proposed method, users can reach their desired images, even when using an ambiguous query text. Experimental results show the proposed method’s effectiveness.

Recallable Question Answering-Based Re-Ranking Considering Semantic Region for Cross-Modal Retrieval

“…Feedback-based re-ranking aims to improve retrieval performance based on user feedback. Several learning-based methods [12], [13], [30], [31] allow users to provide natural language-based feedback on retrieval results. Specifically, a reinforcement learning-based re-ranking method in the fashion domain retrieval is proposed by Guo et al [30].…”

“…Besides, users are required to consider additional natural language-based queries for the re-ranking. Also, as the most relevant method, Yanagi et al [12], [13], [31] proposed a re-ranking method that receives information about objects in the target image. The method proposed in [13], [31] calculates the entropy of each object information based on these existing proportions, and the object information with the largest entropy is used for QA.…”

“…Also, as the most relevant method, Yanagi et al [12], [13], [31] proposed a re-ranking method that receives information about objects in the target image. The method proposed in [13], [31] calculates the entropy of each object information based on these existing proportions, and the object information with the largest entropy is used for QA. Also, the method proposed in [12] uses a trainable fully-connected layer for estimating the object information with the largest entropy, and the estimated information is used for QAbased re-ranking.…”

“…By comparing these baseline methods and our method with them, we confirm that our re-ranking method can improve the initial retrieval performance. Also, basic re-ranking methods [12], [13], [31], [40], [41], [42], [43], [44] were utilized as comparative methods, respectively. By using these various types of re-ranking methods, we confirm that our re-ranking ( f E and f E n ) via the encoders M L (•) and M V (•) following each initial cross-modal retrieval method.…”

“…For example, it is difficult to retrieve the desired image using the text query: "lunch with my friends," from a DB with many similar food scene images. To deal with the problem, we have proposed question answering (QA)-based re-ranking methods that provide questions for clarifying the user's intent to enhance the retrieval performance [12], [13]. The task of QA-based re-ranking is shown in "our task" of Fig.…”