Multi-Scale Speaker Diarization with Neural Affinity Score Fusion

Abstract

Predicting the speaker's identity of short speech segments in human dialogue has been considered one of the most challenging problems in speech signal processing. Speaker representations of short speech segments tend to be unreliable, resulting in poor fidelity of speaker representations in tasks requiring speaker recognition. In this paper, we propose an unconventional method that tackles the trade-off between temporal resolution and the quality of the speaker representations. To find a set of weights that balance the scores from multiple temporal scales of segments, a neural affinity score fusion model is presented. Using the CALLHOME dataset, we show that our proposed multi-scale segmentation and integration approach can achieve a state-of-the-art diarization performance.

From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization

“…To address this, one line of research has combined embeddings extracted with different window sizes and shifts. In [Park et al, 2021], the affinity matrices of three different configurations are processed with an NN and the resulting matrix is used for spectral clustering. In [Kwon et al, 2022], embeddings from different resolutions are paired with vectors that denote which scale they were extracted with (analogously to positional encoding in Transformer models) and they are processed with an attention mechanism to obtain similarities between the embeddings (through the inherent attention weights).…”

Building a Speaker Diarization System: Lessons from VoxSRC 2023

“…These systems involve a dedicated model trained to detect the exact moment when speakers change. To deal with a trade-off between long and short segment lengths, a group of works employs multi-scale segmentation [6,7]. They use multiple scales (segment lengths) and fuse the similarity scores between embeddings obtained from the results of each scale.…”