Lattice-Based Improvements for Voice Triggering Using Graph Neural Networks

Abstract

Voice-triggered smart assistants often rely on detection of a triggerphrase before they start listening for the user request. Mitigation of false triggers is an important aspect of building a privacy-centric non-intrusive smart assistant. In this paper, we address the task of false trigger mitigation (FTM) using a novel approach based on analyzing automatic speech recognition (ASR) lattices using graph neural networks (GNN). The proposed approach uses the fact that decoding lattice of a falsely triggered audio exhibits uncertainties in terms of many alternative paths and unexpected words on the lattice arcs as compared to the lattice of a correctly triggered audio. A pure trigger-phrase detector model doesn't fully utilize the intent of the user speech whereas by using the complete decoding lattice of user audio, we can effectively mitigate speech not intended for the smart assistant. We deploy two variants of GNNs in this paper based on 1) graph convolution layers and 2) self-attention mechanism respectively. Our experiments demonstrate that GNNs are highly accurate in FTM task by mitigating ∼87% of false triggers at 99% true positive rate (TPR). Furthermore, the proposed models are fast to train and efficient in parameter requirements.

Knowledge Transfer for Efficient on-Device False Trigger Mitigation

“…Lattice embeddings are obtained by treating the lattice as a graph and processing it using multiple hidden layers of multi-headed self-attention operation. These embeddings have been shown to be highly informative for FTM task [12,1], but they can be obtained only by running full-fledged ASR on the audio which is expensive to be run on-device and invades user privacy in case of a false trigger. Moreover, the LatticeGNN model needs to be retrained if the distribution of the input lattice features changes due to any changes in the acoustic model, language model or the ASR decoding parameters.…”

“…Other prior approaches for device-directed utterance detection includes various trigger-phrase detection techniques explored in [7,8,9,10,11]. Lattice-based techniques which complement trigger-phrase detection systems have been explored in [12,6,1].…”

“…In these cases, the smart assistant relies on other mechanisms than trigger-phrase detection to determine if the user intends to speak to the device or not. One such mechanism uses self-attention based graph neural networks [1,3] (called "LatticeGNN" hereafter) to extract information from ASR lattices to detect whether the user audio contains spoken content directed towards the device or not. While LatticeGNN ensures that falsely triggered audio is accurately identified, there are limitations in its usage.…”

Complementary Language Model and Parallel Bi-LRNN for False Trigger Mitigation

“…All our experiments are performed on an FTM dataset [8], which is composed of far field usage samples with manual labels of "true trigger" (TT) and "false trigger" (FT) classes. The raw audio data are split into train, cv, dev, and eval sets for the purposes of training, cross-validation, development and evaluation.…”

“…Thus a classifier built on top of the Bi-LRNN is able to mitigate the false trigger cases significantly. A recent work [8] explored the use of graph neural networks (GNN) to encode the decoding lattice, which achieves similar accuracy as the Bi-LRNN representation with more efficient training.…”