Speech Shot Extraction From Broadcast News Videos

Abstract

We propose a method for discriminating between a speech shot and a narrated shot to extract genuine speech shots from a broadcast news video. Speech shots in news videos contain a wealth of multimedia information of the speaker, and could thus be considered valuable as archived material. In order to extract speech shots from news videos, there is an approach that uses the position and size of a face region. However, it is difficult to extract them with only such an approach, since news videos contain non-speech shots where the speaker is not the subject that appears in the screen, namely, narrated shots. To solve this problem, we propose a method to discriminate between a speech shot and a narrated shot in two stages. The first stage of the proposed method directly evaluates the inconsistency between a subject and a speaker based on the co-occurrence between lip motion and voice. The second stage of the proposed method evaluates based on the intra- and inter-shot features that focus on the tendency of speech shots. With the combination of both stages, the proposed method accurately discriminates between a speech shot and a narrated shot. In the experiments, the overall accuracy of speech shots extraction by the proposed method was 0.871. Therefore, we confirmed the effectiveness of the proposed method.

Summarization of Multiple News Videos Considering the Consistency of Audio-Visual Contents

“…To solve this problem, as a special case, we should consider using the speaker's original voice when the selected shot contains a monologue, instead. This could be detected by, for example, Kumagai et al's method [17]. For sentence #3 ( Fig.…”

“…Although in their work, it is shown that this approach is e®ective to some extent, if we do not consider the more high-level visual contents actually present in a scene, it will limit the cases that it could handle properly. Recently, Kumagai et al attempted to detect such inconsistency in news videos based on the relation between audio-visual features [17], but it could only handle monologue (speech) scenes.…”

Report on the analyses and the applications of a large-scale news video archive: NII TV-RECS

“…Thus, we have developed a method that automatically learns and excludes voices of the anchorperson and the reporters according to specific keywords in the CC [12] and a method that learns the correlation of the features between the lip shape and the audio [13], in order to detect monologue scenes. See corresponding references for details of the works.…”