End-To-End Speech Recognition with Joint Dereverberation of Sub-Band Autoregressive Envelopes

Abstract

The end-to-end (E2E) automatic speech recognition (ASR) systems are often required to operate in reverberant conditions, where the long-term sub-band envelopes of the speech are temporally smeared. In this paper, we develop a feature enhancement approach using a neural model operating on sub-band temporal envelopes. The temporal envelopes are modeled using the framework of frequency domain linear prediction (FDLP). The neural enhancement model proposed in this paper performs an envelope gain based enhancement of temporal envelopes. The model architecture consists of a combination of convolutional and long short term memory (LSTM) neural network layers. Further, the envelope dereverberation, feature extraction and acoustic modeling using transformer based E2E ASR can all be jointly optimized for the speech recognition task. We perform E2E speech recognition experiments on the REVERB challenge dataset as well as on the VOiCES dataset. In these experiments, the proposed joint modeling approach yields significant improvements compared to the baseline E2E ASR system (average relative improvements of 21% on the REVERB challenge dataset and about 10% on the VOiCES dataset).

Audio-Visual Multi-Channel Speech Separation, Dereverberation and Recognition

“…However, two issues arise with this pipelined approach: 1) the learning cost function mismatch between speech enhancement frontend and recognition back-end components is not addressed; 2) the artifacts brought by the speech enhancement front-end can lead to ASR performance degradation. To this end, a tight integration of the audio-visual speech separation, dereverberation and recognition components via joint fine-tuning [19], [23], [67], [72], [78]- [82] is considered in this paper. Three finetuning methods are investigated: a) only fine-tuning the backend ASR component using the enhanced speech outputs while the front-end remains unchanged; b) end-to-end jointly finetuning the entire system including the speech enhancement front-end and the recognition back-end components using the ASR cost function; c) end-to-end jointly fine-tuning the entire system using a multi-task criterion interpolation between the speech enhancement and recognition cost functions as follows:…”

“…sys. 3) End-to-end joint fine-tuning of the speech enhancement front-end and recognition back-end is effective in mitigating the impact from spectral artifacts produced in SpecM based dereverberation [82] V. Their WER performance with respect to γ on the LRS2 simulated ("Simu") and replayed ("Replay") test sets are shown in Table VI.…”

“…An improved trade-off between the speech enhancement front-end loss function and ASR accuracy can then be obtained, for example, using multitask learning [67], [78], [79]. To date, such joint speech enhancement front-end and ASR back-end optimization has been only conducted among: a) audio-only speech enhancement and recognition systems using no video input [19], [23], [72], [78], [80]- [82]; or b) audio-visual speech separation and recognition tasks only while not considering speech dereverberation [67], [79]. Hence, there is a pressing need to derive suitable joint optimization methods for a complete audio-visual multi-channel speech separation, dereverberation and recognition system.…”