Distilling the Knowledge of BERT for Sequence-to-Sequence ASR

Futami, Hayato; Inaguma, Hirofumi; Ueno, Sei; Mimura, Mamoru; Sakai, Shinsuke; Kawahara, Tatsuya

doi:10.48550/arxiv.2008.03822

Cited by 8 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous works like shallow fusion [24] and cold fusion [25] aim to combine an auto-regressive LM with a S2S ASR model, which is randomly initialized. To leverage the power of pretrained LM like BERT, many works concentrate on knowledge distillation of BERT for ASR [26,27], while the progress is limited [28]. More recently, some studies aim to integrate the BERT into a non-autoregressive ASR models [28,6].…”

Section: Related Workmentioning

confidence: 99%

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

Deng¹,

Cao²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, self-supervised pretraining has achieved impressive results in end-to-end (E2E) automatic speech recognition (ASR). However, the dominant sequence-to-sequence (S2S) E2E model is still hard to fully utilize the self-supervised pretraining methods because its decoder is conditioned on acoustic representation thus cannot be pretrained separately. In this paper, we propose a pretrained Transformer (Preformer) S2S ASR architecture based on hybrid CTC/attention E2E models to fully utilize the pretrained acoustic models (AMs) and language models (LMs). In our framework, the encoder is initialized with a pretrained AM (wav2vec2.0). The Preformer leverages CTC as an auxiliary task during training and inference. Furthermore, we design a one-cross decoder (OCD), which relaxes the dependence on acoustic representations so that it can be initialized with pretrained LM (DistilGPT2). Experiments are conducted on the AISHELL-1 corpus and achieve a 4.6% character error rate (CER) on the test set. Compared with our vanilla hybrid CTC/attention Transformer baseline, our proposed CTC/attention-based Preformer yields 27% relative CER reduction. To the best of our knowledge, this is the first work to utilize both pretrained AM and LM in a S2S ASR system.

show abstract

Section: Related Workmentioning

confidence: 99%

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

Deng¹,

Cao²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To use the linguistic information from BERT (Devlin et al, 2018) for improving ASR performance, some works (Chiu and Chen, 2021;Shin et al, 2019;Wang and Cho, 2019) use BERT to rerank the N-best hypotheses generated by the ASR model. Besides, knowledge distillation (Futami et al, 2020) (Shin et al, 2019). (b) Cascade methods directly cascade the BERT decoder on the top of the wav2vec 2.0 encoder through Length Alignment module .…”

Section: Speech Recognition With Bertmentioning

confidence: 99%

Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition

Zheng¹,

Xiao²,

Gong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Unifying acoustic and linguistic representation learning has become increasingly crucial to transfer the knowledge learned on the abundance of high-resource language data for low-resource speech recognition. Existing approaches simply cascade pre-trained acoustic and language models to learn the transfer from speech to text. However, how to solve the representation discrepancy of speech and text is unexplored, which hinders the utilization of acoustic and linguistic information. Moreover, previous works simply replace the embedding layer of the pre-trained language model with the acoustic features, which may cause the catastrophic forgetting problem. In this work, we introduce Wav-BERT, a cooperative acoustic and linguistic representation learning method to fuse and utilize the contextual information of speech and text. Specifically, we unify a pre-trained acoustic model (wav2vec 2.0) and a language model (BERT) into an end-to-end trainable framework. A Representation Aggregation Module is designed to aggregate acoustic and linguistic representation, and an Embedding Attention Module is introduced to incorporate acoustic information into BERT, which can effectively facilitate the cooperation of two pre-trained models and thus boost the representation learning. Extensive experiments show that our Wav-BERT significantly outperforms the existing approaches and achieves state-of-the-art performance on low-resource speech recognition.

show abstract

“…The original usage of BERT mainly focused on NLP tasks, ranging from token-level and sequence-level classification tasks, including question answering [9,10], document summarization [11,12], information retrieval [13,14], machine translation [15,16], just to name a few. There has also been attempts to combine BERT in ASR, including rescoring [17,18] or generating soft labels for training [19]. In this section, we review the fundamentals of BERT.…”

Section: Bertmentioning

confidence: 99%

Speech Recognition by Simply Fine-tuning BERT

Huang

Luo

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a simple method for automatic speech recognition (ASR) by fine-tuning BERT, which is a language model (LM) trained on large-scale unlabeled text data and can generate rich contextual representations. Our assumption is that given a history context sequence, a powerful LM can narrow the range of possible choices and the speech signal can be used as a simple clue. Hence, comparing to conventional ASR systems that train a powerful acoustic model (AM) from scratch, we believe that speech recognition is possible by simply fine-tuning a BERT model. As an initial study, we demonstrate the effectiveness of the proposed idea on the AISHELL dataset and show that stacking a very simple AM on top of BERT can yield reasonable performance.

show abstract

Distilling the Knowledge of BERT for Sequence-to-Sequence ASR

Cited by 8 publications

References 22 publications

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition

Speech Recognition by Simply Fine-tuning BERT

Contact Info

Product

Resources

About