Movie Summarization via Sparse Graph Construction

Abstract

We summarize full-length movies by creating shorter videos containing their most informative scenes. We explore the hypothesis that a summary can be created by assembling scenes which are turning points (TPs), i.e., key events in a movie that describe its storyline. We propose a model that identifies TP scenes by building a sparse movie graph that represents relations between scenes and is constructed using multimodal information. According to human judges, the summaries created by our approach are more informative and complete, and receive higher ratings, than the outputs of sequence-based models and general-purpose summarization algorithms. The induced graphs are interpretable, displaying different topology for different movie genres.

Hierarchical3D Adapters for Long Video-to-text Summarization

“…We segment episodes into shots (using PySceneDetect 4 ) and map these to utterances in the corresponding transcript. Specifically, we align the closed captions in the video which are timestamped to the utterances in the transcript using Dynamic Time Warping (DTW; Myers and Rabiner 1981; Papalampidi et al 2021b). We thus create a one-to-many alignment where an utterance corresponds to one or more shots.…”

“…Previous work on video-to-video summarization identifies highlights from YouTube videos, TV shows, or movies (Song et al, 2015;Gygli et al, 2014;De Avila et al, 2011;Papalampidi et al, 2021b). However, in most cases, either the videos are short or the datasets are small with a few hundred examples.…”

“…We build our model on top of a pre-trained sequence-to-sequence architecture (i.e., BART; Lewis et al 2020) fine-tuned on summarization and capable of generating fluent long text. We convert its textual encoder to a multimodal one by adding and tuning adapter layers (Rebuffi et al, 2017;Houlsby et al, 2019), Modality Input Output Datasets text-to-text text short short XSum (Narayan et al, 2018), CNN-DailyMail (Nallapati et al, 2016), NYT (Durrett et al, 2016), Gigaword (Napoles et al, 2012) text long long SamSum (Gliwa et al, 2019), QMSum (Zhong et al, 2021), SummScreen video-to-video vision short short OVP (De Avila et al, 2011), YouTube (De Avila et al, 2011), SumMe (Gygli et al, 2014) vision/text short short TVSum (Song et al, 2015) vision/text(/audio) long long LoL (Fu et al, 2017) TRIPOD+ (Papalampidi et al, 2021b) video-to-text vision long short TACoS (Rohrbach et al, 2014) vision/text/audio short short How2 (Sanabria et al, 2018) vision/text/audio long long SummScreen 3D…”