SpEx+: A Complete Time Domain Speaker Extraction Network

Abstract

Speaker extraction aims to extract the target speech signal from a multi-talker environment given a target speaker's reference speech. We recently proposed a time-domain solution, SpEx, that avoids the phase estimation in frequency-domain approaches. Unfortunately, SpEx is not fully a time-domain solution since it performs time-domain speech encoding for speaker extraction, while taking frequency-domain speaker embedding as the reference. The size of the analysis window for timedomain and the size for frequency-domain input are also different. Such mismatch has an adverse effect on the system performance. To eliminate such mismatch, we propose a complete time-domain speaker extraction solution, that is called SpEx+. Specifically, we tie the weights of two identical speech encoder networks, one for the encoder-extractor-decoder pipeline, another as part of the speaker encoder. Experiments show that the SpEx+ achieves 0.8dB and 2.1dB SDR improvement over the state-of-the-art SpEx baseline, under different and same gender conditions on WSJ0-2mix-extr database respectively.

Muse: Multi-Modal Target Speaker Extraction with Visual Cues

“…As illustrated in Fig. 2(a), speaker encoder consists a stack of three residual blocks followed by an adaptive average pooling layer (Avg-Pool) [10]. The speaker encoder takes a temporal sequence V (t) Ŝ r−1 (t) as input, wherê…”

“…speaker extraction [8]; SpEx/SpEx+ is another successful implementation that trains speaker embedding network jointly with speaker extraction network [7,10].…”

Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanism

“…3. Improved separator with U-Conv blocks level features inside the separation model [6,19] or concatenate the speaker features with the mixture speech representations [8]. However, it is not trivial to find a single optimal layer at which to insert the speaker features.…”

“…An alternative solution to the label permutation problem is to perform target speaker extraction [6][7][8]. In this case, the separation model is biased with information about the identity of the target speaker to extract from the mixture.…”

“…The speaker embedding network can be either jointly trained with the speech extraction model to minimise the enhancement loss or trained on a different task, i.e., a speaker recognition task, to access larger speaker variations [9]. The target speaker embedding is usually inserted into the middle-stage features of the extraction network by using multiplication [7] or concatenation operations [8,10], however, the shared middle-features in the extraction model may not be optimal for both tasks of speaker conditioning and speech reconstruction.…”