Multimodal Speaker Diarization

Abstract

We present a novel probabilistic framework that fuses information coming from the audio and video modality to perform speaker diarization. The proposed framework is a Dynamic Bayesian Network (DBN) that is an extension of a factorial Hidden Markov Model (fHMM) and models the people appearing in an audiovisual recording as multimodal entities that generate observations in the audio stream, the video stream, and the joint audiovisual space. The framework is very robust to different contexts, makes no assumptions about the location of the recording equipment, and does not require labeled training data as it acquires the model parameters using the Expectation Maximization (EM) algorithm. We apply the proposed model to two meeting videos and a news broadcast video, all of which come from publicly available data sets. The results acquired in speaker diarization are in favor of the proposed multimodal framework, which outperforms the single modality analysis results and improves over the state-of-the-art audio-based speaker diarization.

Speaker Diarization: A Review of Recent Research

“…The work presented in [106] integrates audiovisual features for on-line audiovisual speaker diarization using a dynamic Bayesian network (DBN) but tests were limited to discussions with two to three people on two short test scenarios. Another use of DBN, also called factorial HMMs [107], is proposed in [108] as an audiovisual framework. The factorial HMM arises by forming a dynamic Bayesian belief network composed of several layers.…”

Structured exploration of who, what, when, and where in heterogeneous multimedia news sources

“…Speaker diarization seeks to answer the question of "who spoke when," often by clustering detected speech and mapping clusters to names [1]. Recently, [13] explored multimodal speaker diarization using a Dynamic Bayesian Network in both the business meeting and broadcast news videos. Several works extending from [5], have tried to tackle a similar problem using multimodal information for television shows but rely on the a priori presence of fully annotated transcripts that have names mapped to spoken text.…”

Learning Multimodal Temporal Representation for Dubbing Detection in Broadcast Media

“…As the retrieval of information on people in videos is of high interest for users, research efforts have been devoted to unsupervised segmentation of videos into homogeneous segments according to person identity, like speaker diarization [21,17,29], face diarization [5,35], and audio-visual (AV) person diarization [10,25,16,8]. Combined with names extracted from overlaid text, AV person diarization makes it possible to identify people in videos [9].…”