Personalized Speech Enhancement Through Self-Supervised Data Augmentation and Purification

Sivaraman, Aswin; Kim, Sunwoo; Kim, Minje

doi:10.21437/interspeech.2021-1868

Cited by 20 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The trained model, using either contrastive or non-contrastive SSL, is trained to recover premixture sources rather than clean speech, and hence, it requires fine-tuning for the downstream task. Data purification (DP) [126] is later introduced in the pseudo speech enhancement training. Specifically, a separate model is trained to estimate the segmental SNR of the premixture signals, measuring the different importance of the audio frames.…”

Section: Modelmentioning

confidence: 99%

Audio Self-supervised Learning: A Survey

Liu¹,

Mallol-Ragolta²,

Parada-Cabeleiro³

et al. 2022

Preprint

View full text Add to dashboard Cite

Inspired by the humans' cognitive ability to generalise knowledge and skills, Self-Supervised Learning (SSL) targets at discovering general representations from large-scale data without requiring human annotations, which is an expensive and time consuming task. Its success in the fields of computer vision and natural language processing have prompted its recent adoption into the field of audio and speech processing. Comprehensive reviews summarising the knowledge in audio SSL are currently missing. To fill this gap, in the present work, we provide an overview of the SSL methods used for audio and speech processing applications. Herein, we also summarise the empirical works that exploit the audio modality in multimodal SSL frameworks, and the existing suitable benchmarks to evaluate the power of SSL in the computer audition domain. Finally, we discuss some open problems and point out the future directions on the development of audio SSL.

show abstract

Section: Modelmentioning

confidence: 99%

Audio Self-supervised Learning: A Survey

Liu¹,

Mallol-Ragolta²,

Parada-Cabeleiro³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We define PSE as enhancing only the target speaker's voice while removing any other sounds present in the input signal, including interfering speakers and environmental noises. This definition is different than [14], which defines PSE as fine-tuning an unconditional SE model on the target speaker's data without explicitly removing other speakers. The closest work in the literature is [6], which proposed Personalized PercepNet that runs in real-time and can remove the background speakers in addition to the noise.…”

Section: Related Workmentioning

confidence: 99%

Personalized Speech Enhancement: New Models and Comprehensive Evaluation

Eskimez¹,

Yoshioka²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Personalized speech enhancement (PSE) models utilize additional cues, such as speaker embeddings like d-vectors, to remove background noise and interfering speech in real-time and thus improve the speech quality of online video conferencing systems for various acoustic scenarios. In this work, we propose two neural networks for PSE that achieve superior performance to the previously proposed VoiceFilter. In addition, we create test sets that capture a variety of scenarios that users can encounter during video conferencing. Furthermore, we propose a new metric to measure the target speaker over-suppression (TSOS) problem, which was not sufficiently investigated before despite its critical importance in deployment. Besides, we propose multi-task training with a speech recognition back-end. Our results show that the proposed models can yield better speech recognition accuracy, speech intelligibility, and perceptual quality than the baseline models, and the multi-task training can alleviate the TSOS issue in addition to improving the speech recognition accuracy.

show abstract

“…In the mixture invariant training (MixIT) [18], separation models are trained with mixture of mixtures in both unsupervised and semi-supervised setups. An alternative approach to leverage the noisy mixtures is to train the network with pseudo-labels assigned by a pre-trained teacher model, with promising results in singing-voice separation [19,20] and speech enhancement [21,22]. Meanwhile, training with noisy data in TSE is yet to be explored.…”

Section: Introductionmentioning

confidence: 99%

Semi-supervised Time Domain Target Speaker Extraction with Attention

Wang¹,

Giri²,

Venkataramani³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this work, we propose Exformer, a time-domain architecture for target speaker extraction. It consists of a pre-trained speaker embedder network and a separator network based on transformer encoder blocks. We study multiple methods to combine speaker information with the input mixture, and the resulting Exformer architecture obtains superior extraction performance compared to prior time-domain networks. Furthermore, we investigate a two-stage procedure to train the model using mixtures without reference signals upon a pre-trained supervised model. Experimental results show that the proposed semi-supervised learning procedure improves the performance of the supervised baselines.

show abstract

Personalized Speech Enhancement Through Self-Supervised Data Augmentation and Purification

Cited by 20 publications

References 0 publications

Audio Self-supervised Learning: A Survey

Audio Self-supervised Learning: A Survey

Personalized Speech Enhancement: New Models and Comprehensive Evaluation

Semi-supervised Time Domain Target Speaker Extraction with Attention

Contact Info

Product

Resources

About