Movie Question Answering: Remembering the Textual Cues for Layered Visual Contents

Abstract

Movies provide us with a mass of visual content as well as attracting stories. Existing methods have illustrated that understanding movie stories through only visual content is still a hard problem. In this paper, for answering questions about movies, we put forward a Layered Memory Network (LMN) that represents frame-level and clip-level movie content by the Static Word Memory module and the Dynamic Subtitle Memory module, respectively. Particularly, we firstly extract words and sentences from the training movie subtitles. Then the hierarchically formed movie representations, which are learned from LMN, not only encode the correspondence between words and visual content inside frames, but also encode the temporal alignment between sentences and frames inside movie clips. We also extend our LMN model into three variant frameworks to illustrate the good extendable capabilities. We conduct extensive experiments on the MovieQA dataset. With only visual content as inputs, LMN with framelevel representation obtains a large performance improvement. When incorporating subtitles into LMN to form the clip-level representation, we achieve the state-of-the-art performance on the online evaluation task of 'Video+Subtitles'. The good performance successfully demonstrates that the proposed framework of LMN is effective and the hierarchically formed movie representations have good potential for the applications of movie question answering.

SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events

“…Video QA Methods. Extensive studies have been conducted for video QA [61,12,17,23,24,31,49,11,25,60,46,51,8,33,63,66]. Yu et al [61] employed LSTM to encode videos and QA pairs, and adopted an attention mechanism [58].…”

Adversarial Multimodal Network for Movie Question Answering

“…[Na et al, 2017] provides a read-write memory network where the read network and write network consist of multiple convolutional layers, which enable memory read and write operations to have high capacity and flexibility. [Wang et al, 2018a] focuses on the video representation, and puts forward a layered memory network to represent frame-level and clip-level movie content by a static word memory module and another dynamic subtitle memory module respectively.…”

“…Compared with typical multimodal VQA, the problem of cross-modal gap gets more serious in the context of multimodal MovieQA. Firstly, different movies may have different background, sheme, and shooting style, which makes it difficult for learning robust multimodal representations [Wang et al, 2018a]. Secondly, it is common that visual clips and textual subtitles are not aligned in the time coordinate.…”

“…The first one is based on embedding mapping, which learns a visual embedding matrix and takes the sum of visual embedded representations and textual features as the combined multimodal representations [Kim et al, 2017]. The second kind of methods are based on attention mechanism, which attend to the textual memory of the words and sentences for each visual regional features [Tapaswi et al, 2016;Wang et al, 2018a]. The third kind methods are based on compact bilinear pooling [Fukui et al, 2016], which compute a compact analogous outer product between visual features and textual features to get a joint representation [Na et al, 2017].…”

“…The third kind methods are based on compact bilinear pooling [Fukui et al, 2016], which compute a compact analogous outer product between visual features and textual features to get a joint representation [Na et al, 2017]. Among the methods mentioned above, the attention-based models achieve better performance, which project visual representations into textual space [Wang et al, 2018a]. We think this benefits from the textual forms of the combined representations.…”