Cross-modal Joint Prediction and Alignment for Composed Query Image Retrieval

Abstract

In this paper, we focus on the composed query image retrieval task, namely retrieving the target images that are similar to a composed query, in which a modification text is combined with a query image to describe a user's accurate search intention. Previous methods usually focus on learning the joint image-text representations, but rarely consider the intrinsic relationship among the query image, the target image and the modification text. To address this problem, we propose a new cross-modal joint prediction and alignment framework for composed query image retrieval. In our framework, the modification text is regarded as an implicit transformation between the query image and the target image. Motivated by that, not only the combination of the query image and modification text should be similar to the target image, but also the modification text should be predicted according to the query image and the target image. We devote to aligning this relationship by a novel Joint Prediction Module (JPM). Our proposed framework can seamlessly incorporate the JPM into the existing methods to effectively improve the discrimination and robustness of visual and textual representations. The experiments on three public datasets demonstrate the effectiveness of our proposed framework, proving that our proposed JPM can be simply incorporated with the existing methods while effectively improving the performance.

Progressive Learning for Image Retrieval with Hybrid-Modality Queries

“…Subsequently, more and more works focus on this CTI-IR task. The previous works [4,5,12,15,18,19,34,36,39,41] can be categorized into two types. The first type of works [5,15,19,34,36] mainly focus on the multi-modal fusion between image and text queries.…”

“…Previous approaches [5,12,15,18,19,34,36,39] for this task can be categorized into two types. The first type of works [5,15,19,34,36] mainly focus on designing complex components for the multi-modal fusion between text and image queries.…”

“…Wen et al [36] propose to combine local-wise and global-wise composition modules for both local and global modification demands. The second type of works [12,18,39] focus on enhancing the semantic embedding space by combining the image&text-to-image matching and image&image-to-text matching with multi-task learning. For example, Yang et al [39] propose an auxiliary module to align the difference between reference and target images with the modification text by joint prediction.…”

“…The second type of works [12,18,39] focus on enhancing the semantic embedding space by combining the image&text-to-image matching and image&image-to-text matching with multi-task learning. For example, Yang et al [39] propose an auxiliary module to align the difference between reference and target images with the modification text by joint prediction. However, due to the scarcity of supervised data which needs to be in the triplet format as <reference-image, modification-text, target-image>, and the complexity of CTI-IR task which requires both the semantic space learning for target retrieval and cross-modal fusion between hybrid-modality queries, it is hard to effectively learn the complex knowledge together and thus the existing methods in both two types achieve marginally satisfactory retrieval results, with only about 30% of queries retrieving the correct image in the top-10 rank.…”

UIGR: Unified Interactive Garment Retrieval

“…A compositor plays a fundamental role to integrate the textual information with the imagery modality. TGR compositors have been proposed based on various techniques, such as gating mechanism [49], hierarchical attention [7,23,12,20], graph neural network [54,44], joint learning [6,27,44,52,55], ensemble learning [50], style-content modification [29,5] and vision & language pre-training [32].…”