Context Aware Video Caption Generation with Consecutive Differentiable Neural Computer

Abstract

Recent video captioning models aim at describing all events in a long video. However, their event descriptions do not fully exploit the contextual information included in a video because they lack the ability to remember information changes over time. To address this problem, we propose a novel context-aware video captioning model that generates natural language descriptions based on the improved video context understanding. We introduce an external memory, differential neural computer (DNC), to improve video context understanding. DNC naturally learns to use its internal memory for context understanding and also provides contents of its memory as an output for additional connection. By sequentially connecting DNC-based caption models (DNC augmented LSTM) through this memory information, our consecutively connected DNC architecture can understand the context in a video without explicitly searching for event-wise correlation. Our consecutive DNC is sequentially trained with its language model (LSTM) for each video clip to generate context-aware captions with superior quality. In experiments, we demonstrate that our model provides more natural and coherent captions which reflect previous contextual information. Our model also shows superior quantitative performance on video captioning in terms of BLEU (BLEU@4 4.37), METEOR (9.57), and CIDEr-D (28.08).

MFVC: Urban Traffic Scene Video Caption Based on Multimodal Fusion

“…Silvio et al [5] added Inception-Resnet-V2 to extract the motion features of the video and then used soft-attention LSTM (SA-LSTM) as the decoder. Kim et al [6] used CNN to extract 2D features of video frames and then fed the extracted features to a differential neural computer (DNC) to learn the contextual information. The features of video frames are input into a DNC in chronological order, and its memory can store contextual information and fully use contextual information to generate captions.…”

Robustness of Differentiable Neural Computer Using Limited Retention Vector-based Memory Deallocation in Language Model

“…A concatenated vector is used as inputs of the controller D. Two output vectors of the controller are generated: 1) a controller output vector and 2) an interface vector . The controller output vector is equal to the output of a hidden layer in the deep learning model [26]. The interface vector determines the memory address accessed at time-step t to perform the read and write operation [27].…”