Domain Adversarial Training for Accented Speech Recognition

Sun, Sining; Yeh, Ching-Feng; Hwang, Mei‐Yuh; Ostendorf, Mari; Xie, Lei

doi:10.1109/icassp.2018.8462663

Cited by 93 publications

(71 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(b) GAI embedding from F2. proach in spirits, aiming to learn accent-invariant features through gradient reversal [16]. The gradient reversal approach keeps modules of GAI , DAI and ASR model in Fig.…”

Section: Baselinesmentioning

confidence: 99%

Aipnet: Generative Adversarial Pre-Training of Accent-Invariant Networks for End-To-End Speech Recognition

Chen

Zhao-jun

Yeh

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

As one of the major sources in speech variability, accents have posed a grand challenge to the robustness of speech recognition systems. In this paper, our goal is to build a unified end-to-end speech recognition system that generalizes well across accents. For this purpose, we propose a novel pre-training framework AIPNet based on generative adversarial nets (GAN) for accent-invariant representation learning: Accent Invariant Pre-training Networks. We pre-train AIPNet to disentangle accent-invariant and accent-specific characteristics from acoustic features through adversarial training on accented data for which transcriptions are not necessarily available. We further fine-tune AIPNet by connecting the accent-invariant module with an attention-based encoder-decoder model for multiaccent speech recognition. In the experiments, our approach is compared against four baselines including both accent-dependent and accent-independent models. Experimental results on 9 English accents show that the proposed approach outperforms all the baselines by 2.3 ∼ 4.5% relative reduction on average WER when transcriptions are available in all accents and by 1.6 ∼ 6.1% relative reduction when transcriptions are only available in US accent.

show abstract

Section: Baselinesmentioning

confidence: 99%

Aipnet: Generative Adversarial Pre-Training of Accent-Invariant Networks for End-To-End Speech Recognition

Chen

Zhao-jun

Yeh

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [26,27], the authors matched the distributions of the clean and distorted speeches in the feature space, and confirmed that the noise-invariant features were beneficial to robust acoustic models. In [28][29][30][31], speaker-invariant and accent-invariant features were extracted in a similar fashion for speaker recognition and speech recognition. In [32], Domain Separation Network with three network components was used to extract the features.…”

Section: Related Workmentioning

confidence: 99%

Noise Adaptive Speech Enhancement Using Domain Adversarial Training

et al. 2019

View full text Add to dashboard Cite

In this study, we propose a novel noise adaptive speech enhancement (SE) system, which employs a domain adversarial training (DAT) approach to tackle the issue of a noise type mismatch between the training and testing conditions. Such a mismatch is a critical problem in deep-learning-based SE systems. A large mismatch may cause a serious performance degradation to the SE performance. Because we generally use a well-trained SE system to handle various unseen noise types, a noise type mismatch commonly occurs in real-world scenarios. The proposed noise adaptive SE system contains an encoder-decoderbased enhancement model and a domain discriminator model. During adaptation, the DAT approach encourages the encoder to produce noise-invariant features based on the information from the discriminator model and consequentially increases the robustness of the enhancement model to unseen noise types. Herein, we regard stationary noises as the source domain (with the ground truth of clean speech) and non-stationary noises as the target domain (without the ground truth). We evaluated the proposed system on TIMIT sentences. The experiment results show that the proposed noise adaptive SE system successfully provides significant improvements in PESQ (19.0%), SSNR (39.3%), and STOI (27.0%) over the SE system without an adaptation.

show abstract

“…By doing so, it learns invariance to speaker characteristics. This approach has been previously used for speech recognition to learn invariance to noise conditions [9], speaker identity [10,11,12] and accent [13]. The authors in [11], proposed to minimize the senone classification loss, and simultaneously maximize the speaker classification loss.…”

Section: Introductionmentioning

confidence: 99%

To Reverse the Gradient or Not: an Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition

Adi

Zeghidour

Collobert

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Transcribed datasets typically contain speaker identity for each instance in the data. We investigate two ways to incorporate this information during training: Multi-Task Learning and Adversarial Learning. In multi-task learning, the goal is speaker prediction; we expect a performance improvement with this joint training if the two tasks of speech recognition and speaker recognition share a common set of underlying features. In contrast, adversarial learning is a means to learn representations invariant to the speaker. We then expect better performance if this learnt invariance helps generalizing to new speakers. While the two approaches seem natural in the context of speech recognition, they are incompatible because they correspond to opposite gradients back-propagated to the model. In order to better understand the effect of these approaches in terms of error rates, we compare both strategies in controlled settings. Moreover, we explore the use of additional un-transcribed data in a semi-supervised, adversarial learning manner to improve error rates. Our results show that deep models trained on big datasets already develop invariant representations to speakers without any auxiliary loss. When considering adversarial learning and multi-task learning, the impact on the acoustic model seems minor. However, models trained in a semi-supervised manner can improve error-rates.

show abstract

Domain Adversarial Training for Accented Speech Recognition

Cited by 93 publications

References 16 publications

Aipnet: Generative Adversarial Pre-Training of Accent-Invariant Networks for End-To-End Speech Recognition

Aipnet: Generative Adversarial Pre-Training of Accent-Invariant Networks for End-To-End Speech Recognition

Noise Adaptive Speech Enhancement Using Domain Adversarial Training

To Reverse the Gradient or Not: an Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition

Contact Info

Product

Resources

About