Unsupervised Representation Learning with Task-Agnostic Feature Masking for Robust End-to-End Speech Recognition

Abstract

Unsupervised learning-based approaches for training speech vector representations (SVR) have recently been widely applied. While pretrained SVR models excel in relatively clean automatic speech recognition (ASR) tasks, such as those recorded in laboratory environments, they are still insufficient for practical applications with various types of noise, intonation, and dialects. To cope with this problem, we present a novel unsupervised SVR learning method for practical end-to-end ASR models. Our approach involves designing a speech feature masking method to stabilize SVR model learning and improve the performance of the ASR model in a downstream task. By introducing a noise masking strategy into diverse combinations of the time and frequency regions of the spectrogram, the SVR model becomes a robust representation extractor for the ASR model in practical scenarios. In pretraining experiments, we train the SVR model using approximately 18,000 h of Korean speech datasets that included diverse speakers and were recorded in environments with various amounts of noise. The weights of the pretrained SVR extractor are then frozen, and the extracted speech representations are used for ASR model training in a downstream task. The experimental results show that the ASR model using our proposed SVR extractor significantly outperforms conventional methods.

Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

“…v) On-the-fly VR-SBE plus FBK+VR-SBE driven f-LHUC adaptation (Sys.12) not only outperforms the comparable baselines replacing VR-SBE with iVector or xVector (Sys. [10][11], but also gives further WER reductions over VR-SBE adaptation alone by 0.64% abs. (2.03% rel.)…”

“…significant improvement (α = 0.05) obtained over iVector (Sys. 2,10,14,22,27), xVector (Sys. 3,11,15,23,28), or both.…”

“…to 10 ms (Sys. [3][4][5][6][7][8][9][10]. This suggests that VR-SBE features extracted on the fly can instantaneously capture homogeneous characteristics of dysarthric speakers.…”