Direct Acoustics-to-Word Models for English Conversational Speech Recognition

Abstract

Recent work on end-to-end automatic speech recognition (ASR) has shown that the connectionist temporal classification (CTC) loss can be used to convert acoustics to phone or character sequences. Such systems are used with a dictionary and separately-trained Language Model (LM) to produce word sequences. However, they are not truly end-to-end in the sense of mapping acoustics directly to words without an intermediate phone representation. In this paper, we present the first results employing direct acoustics-to-word CTC models on two well-known public benchmark tasks: Switchboard and CallHome. These models do not require an LM or even a decoder at run-time and hence recognize speech with minimal complexity. However, due to the large number of word output units, CTC word models require orders of magnitude more data to train reliably compared to traditional systems. We present some techniques to mitigate this issue. Our CTC word model achieves a word error rate of 13.0%/18.8% on the Hub5-2000 Switchboard/CallHome test sets without any LM or decoder compared with 9.6%/16.0% for phone-based CTC with a 4-gram LM. We also present rescoring results on CTC word model lattices to quantify the performance benefits of a LM, and contrast the performance of word and phone CTC models.

Recognizing Long-Form Speech Using Streaming End-to-End Models

“…4 we compare three sampling strategies: 1) Sample from each domain in Tab. 1 with equal probability (Uniform-Domain); 2) Further divide each domain into subdomains 6 , and sample from each subdomain with equal probability (Uniform-Subdomain); 3) Sample from each domain with probability proportional to the total number of utterances in the domain (Count-Weighted). As can be seen in the table, for E2E ASR models, we find that, contrary to [31], the best strategy is to sample utterances proportional to the amount of training data in each domain.…”

Improved Multi-Stage Training of Online Attention-Based Encoder-Decoder Models

“…In [11], it was shown that using a pre-training strategy improves the generalization capability and hence performance of models in several seq2seq problems such as machine translation and abstractive summarization. Initializing a word-based CTC model with a pre-trained phone-based CTC model was found to be useful in [12]. Similarly, multi-task learning on hierarchical models has also been found to be effective, as in [13].…”

Unsupervised Speaker Adaptation Using Attention-Based Speaker Memory for End-to-End ASR

“…Prior studies on speaker adaptation of E2E systems include appending i-vectors to the acoustic features [8], using speakertransformed features obtained by feature space maximum likelihood linear regression (fMLLR) [9], using GMM-derived features [10], or using a speaker adversarial network [11]. Most of these methods apply adaptation only to the input features.…”