Aligning Salient Objects to Queries: A Multi-modal and Multi-object Image Retrieval Framework

Abstract

In this paper we propose an approach for multi-modal image retrieval in multi-labelled images. A multi-modal deep network architecture is formulated to jointly model sketches and text as input query modalities into a common embedding space, which is then further aligned with the image feature space. Our architecture also relies on a salient object detection through a supervised LSTM-based visual attention model learned from convolutional features. Both the alignment between the queries and the image and the supervision of the attention on the images are obtained by generalizing the Hungarian Algorithm using different loss functions. This permits encoding the object-based features and its alignment with the query irrespective of the availability of the co-occurrence of different objects in the training set. We validate the performance of our approach on standard single/multi-object datasets, showing state-of-the art performance in every dataset.

Localizing Infinity-shaped fishes: Sketch-guided object localization in the wild

“…Sketches can be used to retrieve natural photos [11,21,29,54,57], manga [34], 3D shape [52], video [45] and many more. Specifically, sketch based image retrieval (SBIR) [8] has seen lot of research done to bridge this domain gap between sparse line drawings with dense pixel representations using cross modal deep learning techniques [7]. In these lines, different work has also been conducted on fine-grained SBIR [53], zero-shot SBIR [9], large-scale SBIR [48], and many others.…”