Learning to Inference with Early Exit in the Progressive Speech Enhancement

Abstract

In real scenarios, it is often necessary and significant to control the inference speed of speech enhancement systems under different conditions. To this end, we propose a stage-wise adaptive inference approach with early exit mechanism for progressive speech enhancement. Specifically, in each stage, once the spectral distance between adjacent stages lowers the empirically preset threshold, the inference will terminate and output the estimation, which can effectively accelerate the inference speed. To further improve the performance of existing speech enhancement systems, PL-CRN++ is proposed, which is an improved version over our preliminary work PL-CRN and combines stage recurrent mechanism and complex spectral mapping. Extensive experiments are conducted on the TIMIT corpus, the results demonstrate the superiority of our system over state-of-the-art baselines in terms of PESQ, ESTOI and DNSMOS. Moreover, by adjusting the threshold, we can easily control the inference efficiency while sustaining the system performance.

A Survey on Low-Latency DNN-Based Speech Enhancement

“…The authors claim that for many input samples, shallow representation is already adequate for classification. In [ 102 ], the early-exit mechanism was considered for the speech enhancement task. One aspect of using EEM for speech enhancement is that test examples may vary with SNR.…”