Jointly Modeling Embedding and Translation to Bridge Video and Language

Abstract

Automatically describing video content with natural language is a fundamental challenge of multimedia. Recurrent Neural Networks (RNN), which models sequence dynamics, has attracted increasing attention on visual interpretation. However, most existing approaches generate a word locally with given previous words and the visual content, while the relationship between sentence semantics and visual content is not holistically exploited. As a result, the generated sentences may be contextually correct but the semantics (e.g., subjects, verbs or objects) are not true.This paper presents a novel unified framework, named Long Short-Term Memory with visual-semantic Embedding (LSTM-E), which can simultaneously explore the learning of LSTM and visual-semantic embedding. The former aims to locally maximize the probability of generating the next word given previous words and visual content, while the latter is to create a visual-semantic embedding space for enforcing the relationship between the semantics of the entire sentence and visual content. Our proposed LSTM-E consists of three components: a 2-D and/or 3-D deep convolutional neural networks for learning powerful video representation, a deep RNN for generating sentences, and a joint embedding model for exploring the relationships between visual content and sentence semantics. The experiments on YouTube2Text dataset show that our proposed LSTM-E achieves to-date the best reported performance in generating natural sentences: 45.3% and 31.0% in terms of BLEU@4 and METEOR, respectively. We also demonstrate that LSTM-E is superior in predicting Subject-Verb-Object (SVO) triplets to several state-of-the-art techniques.

Early Embedding and Late Reranking for Video Captioning

“…contrast to previous works [9,15], the concatenation of the two features does not yield any improvement. Hence, we rely on the Googlenet-bu4k feature as the visual input to the sentence generation module.…”

“…The pool5 layer after ReLU is used, resulting in a feature vector of 1,024 dimensions. Observing the increasing popularity of 3-D ConvNets (C3D) for video captioning [9,15], we have also experimented with a C3D model trained by Tran et al on one million sports videos [11]. Though being longer (4,096-dim), the C3D feature is inferior to the Googlenetbu4k feature according to our experiments.…”

“…Following the common practice of applying pre-trained ConvNets for video content analysis [7,9,15], we extract ConvNet features for a given video clip. Frames are uniformly sampled from the clip with an interval of 10 frames.…”

“…State-of-the-art approaches rely on a deep convolutional network with a recurrent neural network [13,15], and emphasize on innovating inside the network architecture [9,16]. We focus on enhancing video captioning, without the need to change internal structures of the networks.…”

Movie Description

“…Later Venugopalan et al (2015c) extended this work to extract CNN features from frames which are max-pooled over time. Pan et al (2016b) propose a framework that consists of a 2-/3-D CNN and LSTM trained jointly with a visual-semantic embedding to ensure better coherence between video and text. Xu et al (2015b) jointly address the language generation and video/language retrieval tasks by learning a joint embedding for a deep video model and a compositional semantic language model.…”