Sparse Mixture of Local Experts for Efficient Speech Enhancement

Sivaraman, Aswin; Kim, Minje

doi:10.21437/interspeech.2020-2989

Cited by 11 publications

(7 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Denoting K to be the number of clusters, one can create K separate SE models trained only to denoise utterances from each disjoint group of similar speakers. As previously shown [12,13], a sparsely active ensemble model is capable of performing zero-shot adaptation because the gating module classifies the test-time noisy input signals into one-of-K groups.…”

Section: Ensemble Modelsmentioning

confidence: 99%

“…MLoE is also extended to recurrent neural network models to learn from the temporal structure of the speech enhancement problem in [11]. While in these methods a local expert is not dedicated to solve a specific sub-problem, in [12] MLoE-based SE system shows substantial improvement by pre-defining two separate partitioning schemes: based on the quality of input signal, i.e., in terms of signalto-noise ratio (SNR), and the gender of the speakers. Moreover, by introducing "sparseness" to the ensemble weights, it performs test-time inference on only one most suitable specialist.…”

Section: Introductionmentioning

confidence: 99%

“…Compared to an ordinary generalist that needs to be very large to achieve a certain level of speech denoising, each specialist can achieve the same quality enhancement results using much smaller architecture. Hence, the sparse ensemble of specialists is claimed as a model compression method in [12]. Zezario et.…”

Section: Introductionmentioning

confidence: 99%

“…a u Z t 2 c z 9 7 6 j n O 6 T v W 6 O d 4 f p q + G + 5 8 D 8 A O y 1 C Z 5 T J 6 S 5 y Q l r 8 k h + U S O y I g w 4 s k P c k E u e 7 + j r W g 3 e r h s j X q r m Q f k i q I n f w E q x b y R < / l a t e x i t > k ⇤ = arg max k pk < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 a y u V N N l P K F L F 8 / 1 z T 7 w N 3 c 0 the test-time signal, i.e. its signal-to-noise (SNR) level, [14,13,12], types of noise sources [15], or the gender [12].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Zero-Shot Personalized Speech Enhancement through Speaker-Informed Model Selection

Sivaraman¹,

Kim²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a novel zero-shot learning approach towards personalized speech enhancement through the use of a sparsely active ensemble model. Optimizing speech denoising systems towards a particular test-time speaker can improve performance and reduce run-time complexity. However, test-time model adaptation may be challenging if collecting data from the test-time speaker is not possible. To this end, we propose using an ensemble model wherein each specialist module denoises noisy utterances from a distinct partition of training set speakers. The gating module inexpensively estimates test-time speaker characteristics in the form of an embedding vector and selects the most appropriate specialist module for denoising the test signal. Grouping the training set speakers into non-overlapping semantically similar groups is non-trivial and ill-defined. To do this, we first train a Siamese network using noisy speech pairs to maximize or minimize the similarity of its output vectors depending on whether the utterances derive from the same speaker or not. Next, we perform k-means clustering on the latent space formed by the averaged embedding vectors per training set speaker. In this way, we designate speaker groups and train specialist modules optimized around partitions of the complete training set. Our experiments show that ensemble models made up of low-capacity specialists can outperform high-capacity generalist models with greater efficiency and improved adaptation towards unseen test-time speakers.

show abstract

Section: Ensemble Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Zero-Shot Personalized Speech Enhancement through Speaker-Informed Model Selection

Sivaraman¹,

Kim²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, ZSL for speech enhancement has not been widely studied. In [17], a mixture of local expert model was introduced as a ZSL solution to test-time adaptation of an SE model. It achieves the adaptation goal by employing a classifier to select the most suitable one out of pre-trained specialist models for a given noisy test signal.…”

Section: Introductionmentioning

confidence: 99%

Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation

Kim¹,

Kim²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In realistic speech enhancement settings for end-user devices, we often encounter only a few speakers and noise types that tend to reoccur in the specific acoustic environment. We propose a novel personalized speech enhancement method to adapt a compact denoising model to the test-time specificity. Our goal in this test-time adaptation is to utilize no clean speech target of the test speaker, thus fulfilling the requirement for zero-shot learning. To complement the lack of clean utterance, we employ the knowledge distillation framework. Instead of the missing clean utterance target, we distill the more advanced denoising results from an overly large teacher model, and use it as the pseudo target to train the small student model. This zero-shot learning procedure circumvents the process of collecting users' clean speech, a process that users are reluctant to comply due to privacy concerns and technical difficulty of recording clean voice. Experiments on various test-time conditions show that the proposed personalization method achieves significant performance gains compared to larger baseline networks trained from a large speaker-and noise-agnostic datasets. In addition, since the compact personalized models can outperform larger general-purpose models, we claim that the proposed method performs model compression with no loss of denoising performance.

show abstract

Speech Enhancement Using Dynamic Learning in Knowledge Distillation via Reinforcement Learning

Chu,

Wu,

2023

IEEE Access

View full text Add to dashboard Cite

In recent years, most of the research on speech enhancement (SE) has applied different strategies to improve performance through deep neural network models. However, as the performance improves, the memory resources and computational requirements of the model also increase, making it difficult to directly apply them to edge computing. Therefore, various model compression and acceleration techniques are desired. This paper proposes a learning method that dynamically uses Knowledge Distillation (KD) to teach a small student model from a large teacher model by considering the learning ratio from the teacher's output and the real target based on reinforcement learning (RL). During the KD learning process, RL is adopted to estimate the learning ratio by considering the reward favoring the hard target (clean speech) or the soft target (the output of the teacher model) during the training of KD. The proposed method results in a more stable training process for the resulting smaller SE model and yields improved performance. In the experiment, we used the TIMIT and CSTR VCTK datasets and evaluated two representative SE models that employ different loss functions. On the TIMIT dataset, when we reduced the number of parameters in the Wave-U-Net student model from 10.3 million to 2.6 million, our method performed better than non-KD models with improvements of 0.05 in PESQ, 0.1 in STOI, and 0.47 in the scale-invariant signal-to-distortion ratio. Moreover, by utilizing prior knowledge from the pre-trained teacher model, our method effectively guided the learning process of the student model, achieving excellent performance even under low SNR conditions. Furthermore, we use Conv-Tasnet to further validate our proposed method. Finally, for ease of comparison, we conducted a comparison on the VCTK dataset as well.

show abstract

Sparse Mixture of Local Experts for Efficient Speech Enhancement

Cited by 11 publications

References 27 publications

Zero-Shot Personalized Speech Enhancement through Speaker-Informed Model Selection

Zero-Shot Personalized Speech Enhancement through Speaker-Informed Model Selection

Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation

Speech Enhancement Using Dynamic Learning in Knowledge Distillation via Reinforcement Learning

Contact Info

Product

Resources

About