Combining Acoustic Embeddings and Decoding Features for End-of-Utterance Detection in Real-Time Far-Field Speech Recognition Systems

Maas, Roland; Rastrow, Ariya; Ma, Chengyuan; Lan, Guitang; Goehner, Kyle; Tiwari, Gautam; Joseph, Shaun N.; Hoffmeister, Björn

doi:10.1109/icassp.2018.8461478

Cited by 25 publications

(24 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The combined feature-based EPD algorithm [39] consists of three parts to detect the EOU exactly by fusing multiple features. They are an acoustic LSTM trained on the acoustic features, the word LSTM trained on the 1-best ASR decoding hypothesis, and the DSFs composed of three types of pause durations, which are described as follows.…”

Section: B Epd Based On Combining Afe We and Dsfsmentioning

confidence: 99%

“…To overcome this disadvantage, the expected pause duration is introduced as the stable feature for the online EPD task since it is obtained by interpolating the pause durations within all active hypotheses [37], [38]. Furthermore, it was observed that the word embedding (WE), which is obtained from the word LSTM [39] trained with the 1-best ASR decoding hypothesis to detect the turn-taking word, can yield the significant performance improvement of acoustic feature embedding (AFE)-based EPD without an actual decoding process, whereas the combination of the AFE, WE, and expected pause durations achieves the stateof-the-art EPD performance. Also, [40] incorporated the EOU symbol into the output within unified recurrent neural networks (RNN) transducer-based ASR system.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

End-to-End Speech Endpoint Detection Utilizing Acoustic and Language Modeling Knowledge for Online Low-Latency Speech Recognition

Hwang

Chang

2020

IEEE Access

View full text Add to dashboard Cite

learning multi-speaker prosody and emotion cloning technology based on a high quality end-to-end model using small amount of data) ABSTRACT Speech endpoint detection (EPD) benefits from the decoder state features (DSFs) of online automatic speech recognition (ASR) system. However, the DSFs are obtained via the ASR decoding process, which can become prohibitively expensive especially in limited-resource scenarios such as the embedded devices. To address this problem, this paper proposes a language model (LM)-based end-of-utterance (EOU) predictor, which is trained to determine the framewise probabilities of the EOU token conditioned on the previous word history obtained from the 1-best decoding hypothesis of the ASR system in an end-to-end manner without an actual decoding process in the test step. Further, a novel end-to-end EPD strategy is presented to incorporate a phonetic embedding (PE)-based acoustic modeling knowledge and the proposed EOU predictor-based language modeling knowledge into an acoustic feature embedding (AFE)-based EPD approach within the recurrent neural networks (RNN)-based EPD framework. The proposed EPD algorithm is built upon the ensemble RNNs, which are independently trained for the three parts, which are the proposed LM-based EOU predictor, AFE-based EPD, and PE-based acoustic model (AM) in accordance with each target. The ensemble RNNs are concatenated at the level of the last hidden layers and then attached into the fully-connected deep neural networks (DNN)-based EPD classifier, which is trained in accordance with the ultimate EPD target. Thereafter, they are jointly retrained at the second step of the DNN training to yield the lower endpoint error. The proposed EPD framework was evaluated in terms of the endpoint accuracy and word error rate for the CHiME-3 and large-scale ASR tasks. The experimental results turn out that the proposed EPD algorithm efficiently outperforms the conventional EPD approaches. INDEX TERMS acoustic model (AM), end-of-turn detection, end-of-utterance (EOU) detection, feature embedding, language model (LM), online speech recognition, pause hesitation, speech endpoint detection (EPD), spoken dialogue system

show abstract

Section: B Epd Based On Combining Afe We and Dsfsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

End-to-End Speech Endpoint Detection Utilizing Acoustic and Language Modeling Knowledge for Online Low-Latency Speech Recognition

Hwang

Chang

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…We conduct our experiments on training data of 1200-hour live data in English collected from the Amazon Echo. Each utterance is handtranscribed and begins with the same wake-up word whose alignment with time is provided by end-point detection [22,23,24,25]. As we have mentioned, while the training data is relatively clean and usually only contains device-directed speech, the test data is more challenging and under mismatched conditions with training data: it may be noisy, may contain background speech 4 , or may even contain no device-directed speech at all.…”

Section: Experimental Settingsmentioning

confidence: 99%

“…Also, as we mentioned in Sec 3.2, multi-task training can be conducted since we know the gold mask for each synthesized utterance. Given the imbalanced mask labels, 25.0% for the "hard" set, then the normalized values would be 1.000 and 5.000 respectively.…”

Section: Mask-based Modelmentioning

confidence: 99%

End-to-end Anchored Speech Recognition

Wang

Fan

Chen

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Voice-controlled house-hold devices, like Amazon Echo or Google Home, face the problem of performing speech recognition of devicedirected speech in the presence of interfering background speech, i.e., background noise and interfering speech from another person or media device in proximity need to be ignored. We propose two end-to-end models to tackle this problem with information extracted from the anchored segment. The anchored segment refers to the wake-up word part of an audio stream, which contains valuable speaker information that can be used to suppress interfering speech and background noise. The first method is called Multi-source Attention where the attention mechanism takes both the speaker information and decoder state into consideration. The second method directly learns a frame-level mask on top of the encoder output. We also explore a multi-task learning setup where we use the ground truth of the mask to guide the learner. Given that audio data with interfering speech is rare in our training data set, we also propose a way to synthesize "noisy" speech from "clean" speech to mitigate the mismatch between training and test data. Our proposed methods show up to 15% relative reduction in WER for Amazon Alexa live data with interfering background speech without significantly degrading on clean speech.

show abstract

“…To the best of our knowledge, this is the first time neural embeddings are being used for token confidence modeling. There has been prior work on using neural embeddings for other tasks such as endpoint detection [10] and device-directed utterance detection [11]. Our objective is to develop a similar neural embedding based framework to improve upon the baseline token confidence model.…”

Section: Introductionmentioning

confidence: 99%

Improving ASR Confidence Scores for Alexa Using Acoustic and Hypothesis Embeddings

et al. 2019

Self Cite

View full text Add to dashboard Cite

In automatic speech recognition, confidence measures provide a quantitative representation used to assess whether a generated hypothesis text is correct or not. For personal assistant devices like Alexa, automatic speech recognition (ASR) errors are inevitable due to the imperfection of today's speech recognition technology. Hence, confidence scores provide an important metric to gauge the correctness of ASR hypothesis text and enable downstream consumers to subsequently initiate appropriate actions. In this work, our aim is to improve the correctness of our confidence scores by enhancing our baseline model architecture with learned features, namely acoustic and 1-best hypothesis embeddings. These embeddings are obtained by training separate networks on acoustic features and ASR 1-best hypothesis respectively. We present an experimental evaluation on a large US English data set showing a 6% relative equal error rate reduction and 13% relative normalized cross-entropy improvement over our baseline system by incorporating these embeddings. We also present a deeper analysis of the embeddings revealing that the acoustic embedding results in a better prediction of insertion errors whereas the 1-best hypothesis embedding helps to better predict substitution errors.

show abstract

Combining Acoustic Embeddings and Decoding Features for End-of-Utterance Detection in Real-Time Far-Field Speech Recognition Systems

Cited by 25 publications

References 12 publications

End-to-End Speech Endpoint Detection Utilizing Acoustic and Language Modeling Knowledge for Online Low-Latency Speech Recognition

End-to-End Speech Endpoint Detection Utilizing Acoustic and Language Modeling Knowledge for Online Low-Latency Speech Recognition

End-to-end Anchored Speech Recognition

Improving ASR Confidence Scores for Alexa Using Acoustic and Hypothesis Embeddings

Contact Info

Product

Resources

About