Semantic Mask for Transformer based End-to-End Speech Recognition

Wang, Chengyi; Wu, Yu; Du, Yujiao; Li, Jinyu; Liu, Shujie; Lu, Liang; Ren, Shuo; Ye, Guoli; Zhao, Shenghui; Zhou, Ming

doi:10.48550/arxiv.1912.03010

Cited by 26 publications

(30 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our large baseline model gives better test-clean and testother WER results than the reported results in the ESPNET Github repository 2 . Also, our large baseline model outperforms a very recent work with semantic masking technique for transformer ASR [34]. With LM fusion, our proposed fine-tuning approach gives the best results for both small and large settings.…”

Section: Resultsmentioning

confidence: 74%

“…[9] focuses on multi-task learning with CTC, and show that transformerbased end-to-end ASR is highly competitive with RNN-based methods. In [34], a semantic mask based regularization method was introduced for transformer-based ASR to force the decoder to learn a better language model. A hybrid transformer model with deep layers and iterated loss was introduced in [31].…”

Section: Relation To Prior Workmentioning

confidence: 99%

See 1 more Smart Citation

Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks

Haidar¹,

Rezagholizadeh²

2021

Preprint

View full text Add to dashboard Cite

Adversarial training of end-to-end (E2E) ASR systems using generative adversarial networks (GAN) has recently been explored for low-resource ASR corpora. GANs help to learn the true data representation through a two-player min-max game. However, training an E2E ASR model using a large ASR corpus with a GAN framework has never been explored, because it might take excessively long time due to high-variance gradient updates and face convergence issues. In this paper, we introduce a novel framework for fine-tuning a pretrained ASR model using the GAN objective where the ASR model acts as a generator and a discriminator tries to distinguish the ASR output from the real data. Since the ASR model is pre-trained, we hypothesize that the ASR model output (soft distribution vectors) helps to get higher scores from the discriminator and makes the task of the discriminator harder within our GAN framework, which in turn improves the performance of the ASR model in the fine-tuning stage. Here, the pre-trained ASR model is fine-tuned adversarially against the discriminator using an additional adversarial loss. Experiments on full LibriSpeech dataset show that our proposed approach outperforms baselines and conventional GAN-based adversarial models.

show abstract

Section: Resultsmentioning

confidence: 74%

Section: Relation To Prior Workmentioning

confidence: 99%

Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks

Haidar¹,

Rezagholizadeh²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Meanwhile, another 5% WER reduction is observed by better initialization (LFA→LASR). We can also see that, even though the separation model is tuned with ASRmatched, the consistent performance improvement still exists when the ASR model is changed to ASR [38] which uses a slightly different feature and architecture. Finally, compared to the state-of-the-art system reported in [17], our model shows a significant performance improvement, reducing the average WER from 9.1% to 5.7%.…”

Section: Architecture Comparisonmentioning

confidence: 81%

“…This model shows WERs of 2.80% and 6.80% on Librispeech test-clean and test-other, respectively. The second ASR is the one developed in [38], which we call ASR [38]. A concatenation of the filter bank and pitch features are used as the input to this model, and it achieves WERs of 2.08% and 4.95% on test-clean and test-other, respectively.…”

Section: Asr Modelmentioning

confidence: 99%

Investigation of Practical Aspects of Single Channel Speech Separation for ASR

Wu,

Chen,

Chen

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Speech separation has been successfully applied as a frontend processing module of conversation transcription systems thanks to its ability to handle overlapped speech and its flexibility to combine with downstream tasks such as automatic speech recognition (ASR). However, a speech separation model often introduces target speech distortion, resulting in a sub-optimum word error rate (WER). In this paper, we describe our efforts to improve the performance of a single channel speech separation system. Specifically, we investigate a two-stage training scheme that firstly applies a feature level optimization criterion for pretraining, followed by an ASR-oriented optimization criterion using an end-to-end (E2E) speech recognition model. Meanwhile, to keep the model light-weight, we introduce a modified teacher-student learning technique for model compression. By combining those approaches, we achieve a absolute average WER improvement of 2.70% and 0.77% using models with less than 10M parameters compared with the previous state-of-theart results on the LibriCSS dataset for utterance-wise evaluation and continuous evaluation, respectively.

show abstract

“…With the emergence of E2E ASR, researchers [7][8][9][10] explore different E2E ASR scenarios and partly focus on the data augmentation and training strategy due to the nature of data hungry and easy over-fitting. For Uyghur speech recognition, existing works mainly focused on problems such as agglutinative characteristics and low resources, often in conventional structure [11][12][13].…”

Section: Introductionmentioning

confidence: 99%

PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition

Ma¹,

Hu²,

Yolwas³

et al. 2021

Preprint

View full text Add to dashboard Cite

Consonant and vowel reduction are often encountered in Uyghur speech, which might cause performance degradation in Uyghur automatic speech recognition (ASR). Our recently proposed learning strategy based on masking, Phone Masking Training (PMT), alleviates the impact of such phenomenon in Uyghur ASR. Although PMT achieves remarkably improvements, there still exists room for further gains due to the granularity mismatch between masking unit of PMT (phoneme) and modeling unit (word-piece). To boost the performance of PMT, we propose multi-modeling unit training (MMUT) architecture fusion with PMT (PM-MMUT). The idea of MMUT framework is to split the Encoder into two parts including acoustic feature sequences to phoneme-level representation (AF-to-PLR) and phoneme-level representation to word-piece-level representation (PLR-to-WPLR). It allows AF-to-PLR to be optimized by an intermediate phoneme-based CTC loss to learn the rich phoneme-level context information brought by PMT. Experimental results on Uyghur ASR show that the proposed approaches improve significantly, outperforming the pure PMT (reduction WER from 24.0 to 23.7 on Read-Test and from 38.4 to 36.8 on Oral-Test respectively). We also conduct experiments on the 960-hour Librispeech benchmark using ESPnet1, which achieves about 10% relative WER on all the test sets without LM fusion comparing with the latest official ESPnet1 pre-trained model.

show abstract

Semantic Mask for Transformer based End-to-End Speech Recognition

Cited by 26 publications

References 14 publications

Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks

Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks

Investigation of Practical Aspects of Single Channel Speech Separation for ASR

PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition

Contact Info

Product

Resources

About