Cross-Layer Similarity Knowledge Distillation for Speech Enhancement

Abstract

Speech enhancement (SE) algorithms based on deep neural networks (DNNs) often encounter challenges of limited hardware resources or strict latency requirements when deployed in realworld scenarios. However, a strong enhancement effect typically requires a large DNN. In this paper, a knowledge distillation framework for SE is proposed to compress the DNN model. We study the strategy of cross-layer connection paths, which fuses multi-level information from the teacher and transfers it to the student. To adapt to the SE task, we propose a frame-level similarity distillation loss. We apply this method to the deep complex convolution recurrent network (DCCRN) and make targeted adjustments. Experimental results show that the proposed method considerably improves the enhancement effect of the compressed DNN and outperforms other distillation methods.

Sub-Band Knowledge Distillation Framework for Speech Enhancement

“…Inspired by previous work [19,20], we address the issue of dimensionality mismatch between teacher and student models by computing similarity-based distillation losses. The method captures and compares the relationship between batch items at each layer output, between teacher and student (Fig.…”

“…The original implementation from [20] involves reshaping X to [b, ctf ] and matrix multiplying it by its transpose X T to obtain the [b, b] symmetric selfsimilarity matrix G. Analogously, this operation can be performed for each t or f dimension independently with resulting G t/f matrices of size [t/f , b, b]. Such an increase in granularity improved the KD performance in [19]. Here, we obtain even more detailed intraactivation Gram matrices by considering each (t, f ) bin separately, resulting in the G tf self-similarity matrix with shape [t, f , b, b].…”

“…The latter was applied in the context of SE in [18] using considerably large, non-causal student models intended for offline applications. In [19], the authors addressed the dimensionality mismatch problem for the causal SE models by using frame-level Similarity Preserving KD [20] (SPKD). SPKD captures the similarity patterns between network activations for different training examples and aims to match those patterns between the student and the frozen pre-trained teacher models.…”

“…SPKD captures the similarity patterns between network activations for different training examples and aims to match those patterns between the student and the frozen pre-trained teacher models. The authors of [19] also introduced fusion blocks, analogous to [21], to distill relationships between consecutive layers.…”

“…Here, we show that the efficacy of conventional KD methods is limited for tiny, causal SE models. To improve distillation efficacy, we first extend the method from [19] by computing SPKD for each bin of the latent representations, corresponding to the time frame (as in [19]) but also the frequency bin of the input, thus providing more resolution for KD loss optimization. The proposed extension outperforms other similarity-based KD methods which we also explore.…”