Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

Abstract

Abstract-Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher-level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color and orientation. Textual or linguistic saliency is extracted from partof-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.

Movie shot selection preserving narrative properties

“…All subjects, having recently watched the 3 movies, were independently shown the skims in a consecutive manner and in random order. As in [20], the scale was graded the following way: poor (0% -40%), fair (40% -60%), good (60% -75%), very good (75% -90%) and excellent (90% -100%).…”

Effective emotion recognition in movie audio tracks

“…Energy Mean and MFCCs (1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12) standard LPCs (0-13) deviation ZCR Spectral Flux Spectral Rolloff Chroma Vector (0-11) Clarity Table 1. Extracted audio features…”

“…Emotion recognition is an active research area with many applications such as human assistive systems [1], autonomous video summarisation [2], diagnosing patients mental illness, monitoring the drivers emotion variations to avoid accidents and helping the manmachine interactions [3]. Emotion recognition systems can also find applications in key event detection tasks [2], affective analysis in music [4] or dialogue management [5].…”

“…Emotion recognition systems can also find applications in key event detection tasks [2], affective analysis in music [4] or dialogue management [5].…”

Video summarization based on Subclass Support Vector Data Description

“…The process of key frame extraction is also known as "key-framing", "story-boarding" or "static video summarization". A video skim is a video of shorter length than the input stream, which is known as "dynamic video summarization" [1], [5], [6], [7], [8], [9].…”