BW-EDA-EEND: streaming END-TO-END Neural Speaker Diarization for a Variable Number of Speakers

Abstract

We present a novel online end-to-end neural diarization system, BW-EDA-EEND, that processes data incrementally for a variable number of speakers. The system is based on the Encoder-Decoder-Attractor (EDA) architecture of Horiguchi et al., but utilizes the incremental Transformer encoder, attending only to its left contexts and using block-level recurrence in the hidden states to carry information from block to block, making the algorithm complexity linear in time. We propose two variants: For unlimited-latency BW-EDA-EEND, which processes inputs in linear time, we show only moderate degradation for up to two speakers using a context size of 10 seconds compared to offline EDA-EEND. With more than two speakers, the accuracy gap between online and offline grows, but the algorithm still outperforms a baseline offline clustering diarization system for one to four speakers with unlimited context size, and shows comparable accuracy with context size of 10 seconds. For limited-latency BW-EDA-EEND, which produces diarization outputs block-by-block as audio arrives, we show accuracy comparable to the offline clustering-based system.

ASR-aware end-to-end neural diarization

“…In future work, we hope to explore similar techniques with a system like [24] that does not require prior knowledge of the number of speakers and allows inference with low latency. Note that stateof-the-art ASR systems like RNN-Transducer [25] cannot be used to extract phone and word-position information, as they output subword units and not phones.…”

Online Neural Diarization of Unlimited Numbers of Speakers

“…This method uses speaker embeddings to convey information between blocks to make the order of output speakers consistent. BW-EDA-EEND [26] replaced the Transformer encoders in EEND with Transformer-XL [51] to extend EEND-EDA [16], [17] to deal with block-wise inputs. In this method, the hidden state embeddings obtained during processing the previous blocks are used to process the current block, thereby solving the speaker permutation ambiguity between blocks.…”

“…In particular, the clustering part requires a separate algorithm to be developed because most of the wellestablished clustering algorithms for speaker diarization [11], [14] cannot be used for online inference. However, end-to-end methods can relatively be easily utilized in online diarization by using a buffer to store the previous input-result pairs [24], [25] or by simply replacing the network architecture with the one that enables online inference [26].…”

From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization

“…First, the 2000 NIST Speaker Recognition Evaluation [34] dataset, usually referred as "Callhome" [35]. We report results on the subset of 2-speaker conversations using the standard Callhome partition 6 . We will refer to the parts as CH1-2spk and CH2-2spk.…”

“…Most works following the EEND principle have focused on improvements on the architecture or modeling. Some by using self-attention layers [4] or conformer layers [5] instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion [6,7] or when more than one microphone is available [8] or by improving the model iteratively using pseudo-labels [9]. Some have used EEND together with more standard approaches by using EEND-inspired models to find overlaps among pairs of speakers in the output of a cascaded system [10] or leveraging EEND's VAD performance by us-ing an external VAD system [11] or combining short duration diarization outputs to produce better whole-utterance diarization [12,13,14,15].…”