A Closer Look at Audio-Visual Multi-Person Speech Recognition and Active Speaker Selection

Abstract

Audio-visual automatic speech recognition is a promising approach to robust ASR under noisy conditions. However, up until recently it had been traditionally studied in isolation assuming the video of a single speaking face matches the audio, and selecting the active speaker at inference time when multiple people are on screen was put aside as a separate problem. As an alternative, recent work has proposed to address the two problems simultaneously with an attention mechanism, baking the speaker selection problem directly into a fully differentiable model. One interesting finding was that the attention indirectly learns the association between the audio and the speaking face even though this correspondence is never explicitly provided at training time. In the present work we further investigate this connection and examine the interplay between the two problems. With experiments involving over 50 thousand hours of public YouTube videos as training data, we first evaluate the accuracy of the attention layer on an active speaker selection task. Secondly, we show under closer scrutiny that an end-to-end model performs at least as well as a considerably larger two-step system that utilizes a hard decision boundary under various noise conditions and number of parallel face tracks.

Best of Both Worlds: Multi-Task Audio-Visual Automatic Speech Recognition and Active Speaker Detection

“…We present a multi-task learning (MTL) [12] setup for a model that can simultaneously perform audio-visual ASR and active speaker detection, improving previous work on multiperson audio-visual ASR. We show that combining the two tasks is enough to significantly improve the performance of the model in the ASD task relative to the baseline in [10,11], while not degrading the ASR performance of the same model trained exclusively for ASR.…”

“…The exact parameters of the Con-vNet can be found on Table 1. This is an important deviation from [11], where blocks of 3D convolutions were used instead. (2+1)D convolutions not only yield better performance, but are also less TPU memory intensive, allowing training with larger batch sizes, which has shown to be particularly important for obtaining lower word error rates.…”

“…We use 8 attention heads, each with dimension of 64, and a model dimension of 1024. This is another important improvement from [11], where a 6-layer BiLSTM encoder was used instead.…”

“…In a realistic setting, however, multiple faces are potentially simultaneously on screen and one must decide which speaker face video to feed to the model. This first step has been traditionally treated as a separate problem [8,9], but recent work has shown that we can obtain this association with an end-to-end model employing an attention mechanism over the candidate faces [10,11].…”

“…As a side effect, this attention mechanism implicitly captures the correspondence between the audio and the video of the active speaker and, thus, can also be used as an active speaker detection (ASD) model in addition to ASR. However, while interesting in itself, despite having nontrivial accuracy, the implicit association doesn't perform as well as when the attention is trained explicitly for active speaker detection (for a detailed study, see [11]). ASD essentially pro-1 obraga@google.com vides a strong signal for diarization, and high ASD accuracy as a side-product is a compelling reason to include the visual signal in ASR models.…”