A Deliberation-Based Joint Acoustic and Text Decoder

Abstract

We propose a new two-pass E2E speech recognition model that improves ASR performance by training on a combination of paired data and unpaired text data. Previously, the joint acoustic and text decoder (JATD) has shown promising results through the use of text data during model training and the recently introduced deliberation architecture has reduced recognition errors by leveraging first-pass decoding results. Our method, dubbed Deliberation-JATD, combines the spelling correcting abilities of deliberation with JATD's use of unpaired text data to further improve performance. The proposed model produces substantial gains across multiple test sets, especially those focused on rare words, where it reduces word error rate (WER) by between 12% and 22.5% relative. This is done without increasing model size or requiring multi-stage training, making Deliberation-JATD an efficient candidate for on-device applications.

Improving Deliberation by Text-Only and Semi-Supervised Training

“…Text-only data can also be converted to TTS utterances for training the whole deliberation decoder. We employ JATD training [11] and scale it up using text data sampled from multiple domains, i.e., 51M, 20M, 1.6M, 0.6M, and 11M text sentences from Maps, News, Play, Search and YouTube domains, respectively. In comparison, [11] uses only 4.6M samples from the Maps domain.…”

“…While LM relies on only text hypotheses for rescoring, deliberation models have been recently proposed for second-pass rescoring using both text hypotheses and audio [9,10]. Compared to LM training, there has been few attempts at incorporating widely available text-only or audio-only data in deliberation (see [11]). In this work, we research various ways to utilize large-scale textonly and semi-supervised data for deliberation training.…”

“…Instead of training external modules such as LMs, several recent studies incorporate text-only data into supervised training to jointly train E2E models [16,11,17,18,19,20]. For example, text-only data has been used to train speech encoders [16,18].…”

“…On the other hand, ASR decoders have also been modified for text-only training. [11] extends a joint acoustic and text decoder (JATD) from the Listen, Attend and Spell (LAS) [21] to a deliberation decoder, and uses textonly data (or synthesized utterances) to train the decoder with fixed context vectors. [17] modifies the transformer decoder to have only self-attention (except for the last layer) so they can be trained by text-only data.…”

“…Our results show that pretraining a conformer text encoder with large enough size significantly improves recognition for both Voice Search and long-tail words. In addition to the text encoder, we also synthesize large-scale text-only data (84M) to TTS utterances in training the deliberation decoder using JATD [11]. Third, since deliberation attends to encoded audio, we perform large-scale semi-supervised training using 500M unlabeled speech utterances from Google Voice Search domain and transcribed using a conventional model.…”