Dual Encoding for Zero-Example Video Retrieval

Abstract

This paper attacks the challenging problem of zeroexample video retrieval. In such a retrieval paradigm, an end user searches for unlabeled videos by ad-hoc queries described in natural language text with no visual example provided. Given videos as sequences of frames and queries as sequences of words, an effective sequence-to-sequence cross-modal matching is required. The majority of existing methods are concept based, extracting relevant concepts from queries and videos and accordingly establishing associations between the two modalities. In contrast, this paper takes a concept-free approach, proposing a dual deep encoding network that encodes videos and queries into powerful dense representations of their own. Dual encoding is conceptually simple, practically effective and endto-end. As experiments on three benchmarks, i.e. MSR-VTT, TRECVID 2016 and 2017 Ad-hoc Video Search show, the proposed solution establishes a new state-of-the-art for zero-example video retrieval.

Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings

“…These learning-to-rank approaches have been generalised to two or more modalities. Standard examples include building a joint embedding for images and text [11,36], videos and audio [33] and, more related to our work, for videos and action labels [15], videos and text [8,14,40] or some of those combined [25,24,22]. Representing text.…”

“…Early works in image-to-text crossmodal retrieval [9,11,36] used TF-IDF as a weighted bagof-words model for text representations (either from a word embedding model or one-hot vectors) in order to aggregate variable length text captions into a single fixed sized representation. With the advent of neural networks, works shifted to use RNNs, Gated Recurrent Units (GRU) or Long Short-Term Memory (LSTM) units to extract textual features [8] or to use these models within the embedding network [15,18,24,25,34] for both modalities. Action embedding and retrieval.…”

“…We select MSR-VTT [39] as a public dataset for general video retrieval. Originally used for video captioning, this large-scale video understanding dataset is increasingly evaluated for video-to-text and text-to-video retrieval [8,22,24,41,23]. We follow the code and setup of [22] using the same train/test split that includes 7,656 training videos each with 20 different captions describing the scene and 1000 test videos with one caption per video.…”

“…This approach has a number of advantages over training a single embedding space as is standardly done [7,8,15,22,24]. Firstly, this process builds different embeddings that can be seen as different views of the data, which contribute to the final goal in a collaborative manner.…”

A Graph-Based Framework to Bridge Movies and Synopses

“…To retrieve a video with natural language queries, the main challenge is the gap between two different modals. Visual Semantic Embedding (VSE) [9,7], a widely adopted approach in video retrieval [38,18,37,6,35], tries to tackle this problem by embedding multi-modal information into a common space. JSF proposed in [37] learns matching kernels based on fea-ture sequence fusion.…”