Self-Attentional Models for Lattice Inputs

Abstract

Lattices are an efficient and effective method to encode ambiguity of upstream systems in natural language processing tasks, for example to compactly capture multiple speech recognition hypotheses, or to represent multiple linguistic analyses. Previous work has extended recurrent neural networks to model lattice inputs and achieved improvements in various tasks, but these models suffer from very slow computation speeds. This paper extends the recently proposed paradigm of self-attention to handle lattice inputs. Self-attention is a sequence modeling technique that relates inputs to one another by computing pairwise similarities and has gained popularity for both its strong results and its computational efficiency. To extend such models to handle lattices, we introduce probabilistic reachability masks that incorporate lattice structure into the model and support lattice scores if available. We also propose a method for adapting positional embeddings to lattice structures. We apply the proposed model to a speech translation task and find that it outperforms all examined baselines while being much faster to compute than previous neural lattice models during both training and inference.

Lattice-Based Improvements for Voice Triggering Using Graph Neural Networks

“…While the lattice RNNs perform similar to the GNN-based FTM, we found that they are often inconvenient due to increased training time as compared to the GNNs (∼8min/epoch for RNN training v/s ∼1.5min/epoch for GNN training in our experiments). As observed in [13], unique connections in lattices inhibit efficient batching of training examples during RNN training. On the other hand, in GNNs, recurrent computations are replaced by graph convolution operations and multiple lattices of different sizes and structures can be efficiently batched together using zero padding resulting in substantial training speed-up.…”

“…Self-attention graph neural networks (SAGNN) [14,13,11,12] model the relationship between the nodes of a graph using the self-attention mechanism instead of the predefined edges in the graph. Instead of using a fixed adjacency matrix as in GCNs, this approach uses the inner-product of the feature vectors of the lattice arcs to compute their relevance to each other.…”

“…Lattice-based FTM has been successfully explored in [7,8] and has shown that confusion in ASR lattices acts as a strong evidence of false alarm. Prior work on GNNs has been comprehensively summarized in [9,10,11,12]; and [13] demonstrates use of self-attention based GNNs on lattice inputs for a machine translation task.…”

Discourse Self-Attention for Discourse Element Identification in Argumentative Student Essays

“…Mihaylov and Frank (2019) proposed a discourse-aware selfattention encoder for reading comprehension on narrative texts, where event chains, discourse relations and coreference relations are used for connecting sentences. Self-attention can be also extended to 2d-dimensions for image processing (Parmar et al, 2018) and lattice inputs (Sperber et al, 2019).…”

Direct Segmentation Models for Streaming Speech Translation

“…In the cascade approach, an ASR system transcribes the input speech signal, and this is fed to a downstream MT system that carries out the translation. The provided input to the MT step can be the 1-best hypothesis, but also n-best lists (Ng et al, 2016) or even lattices (Matusov and Ney, 2011;Sperber et al, 2019). Additional techniques can also be used to improve the performance of the pipeline by better adapting the MT system to the expected input, such as training with transcribed text (Peitz et al, 2012) or chunking (Sperber et al, 2017).…”