Improving CTC-based speech recognition via knowledge transferring from pre-trained language models

Deng, Keqi; Cao, Songjun; Zhang, Yike; Ma, Lei; Cheng, Gaofeng; Xu, Jianzhong; Zhang, Pengyuan

doi:10.48550/arxiv.2203.03582

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, neural network based FA strategy like NeuFA [?] and CIF [6,4] is proposed. Even Connectionist Temporal Classification (CTC) can be generally regarded as an FA strategy.…”

Section: Forced Alignment Strategiesmentioning

confidence: 99%

“…For the concern of reducing the information loss of BERT, which contains more information than wave models, the parameters of BERT are fixed. Inspired by the work [4], we use the serial CIF [6] mechanism to achieve the monotonic alignment between the speech and text modalities.…”

Section: Alignment Between Two Modalsmentioning

confidence: 99%

“…Regarding L i as the learning target of the Â, We calculate the loss between Â and L i . The work [4] uses the equation below to calculate the loss between two modals for alignment:…”

Section: Alignment Between Two Modalsmentioning

confidence: 99%

See 2 more Smart Citations

WaBERT: A Low-resource End-to-end Model for Spoken Language Understanding and Speech-to-BERT Alignment

Ye¹,

Song²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Historically lower-level tasks such as automatic speech recognition (ASR) and speaker identification are the main focus in the speech field. Interest has been growing in higher-level spoken language understanding (SLU) tasks recently, like sentiment analysis (SA). However, improving performances on SLU tasks remains a big challenge. Basically, there are two main methods for SLU tasks: (1) Two-stage method, which uses a speech model to transfer speech to text, then uses a language model to get the results of downstream tasks; (2) One-stage method, which just fine-tunes a pre-trained speech model to fit in the downstream tasks. The first method loses emotional cues such as intonation, and causes recognition errors during ASR process, and the second one lacks necessary language knowledge. In this paper, we propose the Wave BERT (WaBERT), a novel end-to-end model combining the speech model and the language model for SLU tasks. WaBERT is based on the pre-trained speech and language model, hence training from scratch is not needed. We also set most parameters of WaBERT frozen during training. By introducing WaBERT, audio-specific information and language knowledge are integrated in the short-time and low-resource training process to improve results on the dev dataset of SLUE SA tasks by 1.15% of recall score and 0.82% of F1 score. Additionally, we modify the serial Continuous Integrate-and-Fire (CIF) mechanism to achieve the monotonic alignment between the speech and text modalities.

show abstract

“…Also, neural network based FA strategy like NeuFA [?] and CIF [6,4] is proposed. Even Connectionist Temporal Classification (CTC) can be generally regarded as an FA strategy.…”

Section: Forced Alignment Strategiesmentioning

confidence: 99%

Section: Alignment Between Two Modalsmentioning

confidence: 99%