Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Abstract

Speaker adaptation techniques provide a powerful solution to customise automatic speech recognition (ASR) systems for individual users. Practical application of unsupervised modelbased speaker adaptation techniques to data intensive end-to-end ASR systems is hindered by the scarcity of speaker-level data and performance sensitivity to transcription errors. To address these issues, a set of compact and data efficient speaker-dependent (SD) parameter representations are used to facilitate both speaker adaptive training and test-time unsupervised speaker adaptation of state-of-the-art Conformer ASR systems. The sensitivity to supervision quality is reduced using a confidence score-based selection of the less erroneous subset of speaker-level adaptation data. Two lightweight confidence score estimation modules are proposed to produce more reliable confidence scores. The data sparsity issue, which is exacerbated by data selection, is addressed by modelling the SD parameter uncertainty using Bayesian learning. Experiments on the benchmark 300-hour Switchboard and the 233-hour AMI datasets suggest that the proposed confidence score-based adaptation schemes consistently outperformed the baseline speaker-independent (SI) Conformer model and conventional non-Bayesian, point estimate-based adaptation using no speaker data selection. Similar consistent performance improvements were retained after external Transformer and LSTM language model rescoring. In particular, on the 300-hour Switchboard corpus, statistically significant WER reductions of 1.0%, 1.3%, and 1.4% absolute (9.5%, 10.9%, and 11.3% relative) were obtained over the baseline SI Conformer on the NIST Hub5'00, RT02, and RT03 evaluation sets respectively. Similar WER reductions of 2.7% and 3.3% absolute (8.9% and 10.2% relative) were also obtained on the AMI development and evaluation sets.

Identity of university Chinese heritage language learners in Hong Kong

“…These include, but not limited to: 1) auxiliary speaker embedding based approaches [14][15][16][17][18], e.g. iVector [14] and xVector [15]; 2) feature transformation based methods, e.g., feature-space MLLR [19]; and 3) model-based methods [20][21][22][23] that estimate speaker dependent (SD) adapter parameters implemented as, e.g. learning hidden unit contributions (LHUC) [21], during speaker adaptive training (SAT) and test-time unsupervised adaptation [22,23].…”

“…3) This paper presents the first investigation of the complete incorporation of speaker features into all the components of a complete end-to-end audio-visual multichannel speech separation and recognition system. In contrast, prior researches consider speaker adaptation of either the speech separation front-end [12,[24][25][26][27][28][29][30][31][32] alone, or the speech recognition backend [14,[16][17][18][19][21][22][23] only.…”

“…In contrast, the related prior studies only focus on less practical enrollment-based adaptation techniques[24][25][26][27][28][29][30][31][32] that require clean speech samples to be explicitly recorded at the onset of user personalization.3) This paper presents the first investigation of the complete incorporation of speaker features into all the components of a complete end-to-end audio-visual multichannel speech separation and recognition system. In contrast, prior researches consider speaker adaptation of either the speech separation front-end[12,[24][25][26][27][28][29][30][31][32] alone, or the speech recognition backend[14,[16][17][18][19][21][22][23] only.…”