Cross-Modal Supervision for Learning Active Speaker Detection in Video

Abstract

Abstract. In this paper, we show how to use audio to supervise the learning of active speaker detection in video. Voice Activity Detection (VAD) guides the learning of the vision-based classifier in a weakly supervised manner. The classifier uses spatio-temporal features to encode upper body motion -facial expressions and gesticulations associated with speaking. We further improve a generic model for active speaker detection by learning person specific models. Finally, we demonstrate the online adaptation of generic models learnt on one dataset, to previously unseen people in a new dataset, again using audio (VAD) for weak supervision. The use of temporal continuity overcomes the lack of clean training data. We are the first to present an active speaker detection system that learns on one audio-visual dataset and automatically adapts to speakers in a new dataset. This work can be seen as an example of how the availability of multi-modal data allows us to learn a model without the need for supervision, by transferring knowledge from one modality to another.

Active speaker detection with audio-visual co-training

“…In this paper, we use the above video-based person-specific active speaker detection models to train personalized audio voice models. This further improves the performance of the detection of active speakers in the dataset used by [3], to almost 100%.…”

“…We use Improved Trajectory (IT) features, spatio-temporal features originally used for action recognition [20], and adapted by [2,3] for active speaker detection. These features are a concatenation of Histogram of Oriented Gradients (HoG), Histogram of Optical Flow (HoF) and Motion Boundary Histogram (MBH) features calculated around feature points tracked over a sequence of 15 frames.…”

“…The use of spatio-temporal features was found to outperform lipmotion to detect speaking [4,7,17], especially in videos with multiple speakers in the scene that lack adequate resolution to discern the motion of individual lips. In further work, Chakravarty et al [3] demonstrated that a generic visual Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.…”

“…VAD was subsequently used to modify and adapt this generic visual model to individual speakers [3]. These individual visual active speaker detection models are used in this work, to learn person-specific audio voice models, to achieve almost perfect classification results.…”

“…With this, we propose a method to close the loop in active speaker detection. Earlier work [2,3] demonstrated the ability of using audio to supervise video. In this paper, we show the reverse, that the learnt video models are capable of successfully supervising the training of audio voice models, thus demonstrating co-training.…”