End-To-End Speaker Segmentation for Overlap-Aware Resegmentation

Abstract

Speaker segmentation consists in partitioning a conversation between one or more speakers into speaker turns. Usually addressed as the late combination of three sub-tasks (voice activity detection, speaker change detection, and overlapped speech detection), we propose to train an end-to-end segmentation model that does it directly. Inspired by the original end-to-end neural speaker diarization approach (EEND), the task is modeled as a multi-label classification problem using permutation-invariant training. The main difference is that our model operates on short audio chunks (5 seconds) but at a much higher temporal resolution (every 16ms). Experiments on multiple speaker diarization datasets conclude that our model can be used with great success on both voice activity detection and overlapped speech detection. Our proposed model can also be used as a post-processing step, to detect and correctly assign overlapped speech regions. Relative diarization error rate improvement over the best considered baseline (VBx) reaches 17% on AMI, 13% on DIHARD 3, and 13% on VoxConverse.

Online Neural Diarization of Unlimited Numbers of Speakers

“…The combination of an end-to-end approach and clustering is a promising direction to solve the problem of the limitation of the number of speakers. For example, EEND as postprocessing [23] and overlap-aware resegmentation [13] use EEND to refine the results obtained with cascaded diarization systems. The initial results are based on clustering of speaker embeddings; hence, the number of output speakers can be arbitrary.…”

“…For evaluating offline diarization, we utilized several cascaded methods [13], [14], [22], [69] and end-to-end methods [16], [17], [29], [32] for comparison. For evaluating online diarization, we used FW-STB with EEND-EDA based on four-stacked Transformers [25].…”

“…In offline diarization, EEND-GLA-Small and EEND-GLA-Large improved the DERs from EEND-EDA, especially when the number of speakers was higher than four. Compared with the cascaded method [72] or the cascaded method incorporated with EEND for post-processing [13], EEND-GLA-Large performed on par with them when the § The values are from the original FW-STB paper [25]. number of speakers was low, but not when the number of speakers was high.…”

Magnitude-aware Probabilistic Speaker Embeddings

“…We used a simple speaker diarization pipeline including the following steps: voice activity detection (VAD), overlapped speech detection (OSD; both from [38,39]), fixedlength segmentation, clustering, and post-processing. The postprocessing includes merging the adjacent sub-segments from the same speaker and distributing the overlapped segments equally among the adjacent segments with different speakers.…”

Collar-aware Training for Streaming Speaker Change Detection in Broadcast Speech

“…We also compare to the neural speaker segmentation method implemented in pyannote.audio [27] that performs joint voice activity detection, speaker segmentation and overlapped speech detection. Similarly to the original EEND approach [28], here speaker segmentation is modeled as a multi-label classification problem using permutation-invariant training.…”