Few-Shot Image and Sentence Matching via Gated Visual-Semantic Embedding

Abstract

Although image and sentence matching has been widely studied, its intrinsic few-shot problem is commonly ignored, which has become a bottleneck for further performance improvement. In this work, we focus on this challenging problem of few-shot image and sentence matching, and propose a Gated Visual-Semantic Embedding (GVSE) model to deal with it. The model consists of three corporative modules in terms of uncommon VSE, common VSE, and gated metric fusion. The uncommon VSE exploits external auxiliary resources to extract generic features for representing uncommon instances and words in images and sentences, and then integrates them by modeling their semantic relation to obtain global representations for association analysis. To better model other common instances and words in rest content of images and sentences, the common VSE learns their discriminative representations directly from scratch. After obtaining two similarity metrics from the two VSE modules with different advantages, the gated metric fusion module adaptively fuses them by automatically balancing their relative importance. Based on the fused metric, we perform extensive experiments in terms of few-shot and conventional image and sentence matching, and demonstrate the effectiveness of the proposed model by achieving the state-of-the-art results on two public benchmark datasets.

Target-Oriented Deformation of Visual-Semantic Embedding Space

“…Recent studies on image-caption retrieval have employed cross-model attentions to pay attention to concepts shared by a query and a target [11], [12], [16]- [19]. Crossmodal attentions are performed at an early phase over multiple cropped regions of an image and words of a caption.…”

“…Cutting-edge methods for image-caption retrieval sometimes employ an object detector and a cross-modal attention [11], [12], [16]- [19]. A pretrained object detector crops multiple subregions in an image, and then a cross-modal attention is performed over the cropped regions and words in a caption.…”

“…Moreover, recent studies involving SCAN [11], GVSE [19], and VSRN [50] reported the results of a twomodel ensemble. In this experimental setting, TOD-Net also improves performance significantly.…”

“…By that means, TOD-Net provides new similarities between entities under a condition while preserving the topological relations between them. • Unlike existing methods based on an object detector and a cross-modal attention [11], [12], [16]- [19], TOD-Net is applied to an embedding space and improves the retrieval performance even when using a single-image encoder. This fact indicates that a singleimage encoder already extracts detailed concepts from an image or caption but encounters a difficulty in expressing their relations in the embedding space.…”

Event-Driven Network for Cross-Modal Retrieval

“…Existing work has focused on the matching between text words and image entities, and retrieves them by capturing the co-occurrence relationship between certain words in a text and certain entities in a image [7]. However, when the text is long with multiple sentences or even paragraphs (e.g.…”