Automatic multi-modal dialogue scene indexing

Abstract

An automatic algorithm for indexing dialogue scenes in multimedia content is proposed. The content is segmented into dialogue scenes using the state transitions of a hidden Markov model (HMM). Each shot is classified using both audio and visual information to determine the state/scene transitions for this model. Face detection and silence/speech/music classification are the basic tools which are utilized to index the scenes. While face information is extracted after applying some heuristics to skin-colored regions, audio analysis is achieved by examining signal energy, periodicity and zero crossing rate (ZCR) of the audio waveform. The simulation results show the possibility of automatically indexing the dialogues using the proposed algorithm.

Dynamic video summarization with content analysis

“…In such dialogue region, frames occur repetitively, and we could use dialogue detection to compress such dialogue frames. The method proposed in [1] is selected to detect dialogue frames.…”

Movie Analysis with Emphasis to Dialogue and Action Scene Detection

“…The false hit rate determines the ratio of falsely detected scene boundaries to the number of all shot boundaries. Finally, Alatan et al [2,3,4] employ the shot accuracy measure, which is defined as the ratio of correct shot assignments to the total number of shots.…”

“…Some methods are extensions of those described in Section 2.4, incorporating the information contained in both the video and the audio channels. The techniques for audiovisual dialogue and action scene detection are classified as deterministic [10,26,29,53] and probabilistic [4,2,3,28,50], like in Section 2.4. While the deterministic methods usually cluster consecutive shots by utilizing appropriate measures, most probabilistic approaches use HMMs representing the semantic events in their states.…”

“…An approach for multi-modal dialogue detection using HMMs has been proposed by Alatan et al [4,2,3]. Each shot is classified into speech, silence, or music based on the audio content and at the same time face occurrences and location changes are detected by analyzing the video content.…”