Exploring Attention Mechanism for Acoustic-based Classification of Speech Utterances into System-directed and Non-system-directed

Abstract

Voice controlled virtual assistants (VAs) are now available in smartphones, cars, and standalone devices in homes. In most cases, the user needs to first "wake-up" the VA by saying a particular word/phrase every time he or she wants the VA to do something. Eliminating the need for saying the wake-up word for every interaction could improve the user experience. This would require the VA to have the capability to detect the speech that is being directed at it and respond accordingly. In other words, the challenge is to distinguish between system-directed and non-system-directed speech utterances. In this paper, we present a number of neural network architectures for tackling this classification problem based on using only acoustic features. These architectures are based on using convolutional, recurrent and feed-forward layers. In addition, we investigate the use of an attention mechanism applied to the output of the convolutional and the recurrent layers. It is shown that incorporating the proposed attention mechanism into the models always leads to significant improvement in classification accuracy. The best model achieved equal error rates of 16.25% and 15.62% on two distinct realistic datasets.

Improving Identification of System-Directed Speech Utterances by Deep Learning of ASR-Based Word Embeddings and Confidence Metrics

“…This paper continues our previous work in [1] where only acoustic features were used for utterance classification. Specifically, we present two methods that incorporate non-acoustic information into our models to improve upon our previous acoustic-only-based performance ; the first incorporates ASR decoder features in addition to the usual acoustic features, while the second further adds word embeddings as inputs to the final classification stage of the model.…”

“…Approaches for the classification of utterances into systemand non-system-directed ones typically use acoustic features extracted from the speech signal, e.g., [1,2,3,4,5]. Previous works [1,6,7] also show that using an attention mechanism *Author performed research herein as part of an internship-partnership program between Mila and Nuance. combined with a BiLSTM network can improve classification performance.…”

“…combined with a BiLSTM network can improve classification performance. Our previous work in [1] employs such a BiLSTM with attention, with acoustic features as inputs.…”

Agent Productivity Modeling in a Call Center Domain Using Attentive Convolutional Neural Networks

“…The CNNs or the BiLSTMs layers generate a sequence of vectors for the classification process [ 32 ]. The attention layer is used to convert the sequence of vectors (frames) into a context vector, which attends some parts of the input sequence [ 33 , 34 ]. Figure 2 illustrates the role of the attention layer in our approach.…”

A Multimodal Approach to Improve Performance Evaluation of Call Center Agent

“…The sequence of vectors (frames) produced from CNN or LSTM and forwarded to the attention layer to convert them into a context vector [ 23 , 28 , 29 ]. The attention weight are forwarded to Softmax function at time t to generate the probability of the frame out of one to the remaining frames in the same speech segment.…”