Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing

Abstract

An effective recipe for building seq2seq, nonautoregressive, task-oriented parsers to map utterances to semantic frames proceeds in three steps: encoding an utterance x, predicting a frame's length |y|, and decoding a |y|sized frame with utterance and ontology tokens. Though empirically strong, these models are typically bottlenecked by length prediction, as even small inaccuracies change the syntactic and semantic characteristics of resulting frames. In our work, we propose span pointer networks, non-autoregressive parsers which shift the decoding task from text generation to span prediction; that is, when imputing utterance spans into frame slots, our model produces endpoints (e.g., [i, j]) as opposed to text (e.g., "6pm"). This natural quantization of the output space also provides consistency in the length prediction task, allowing our length predictor to be responsible for frame syntax and the decoder for frame syntax, creating a coarse-to-fine model. We evaluate our approach on several task-oriented semantic parsing datasets. Notably, we bridge the quality gap between non-autogressive and autoregressive parsers, achieving 87 EM on TOPv2 (Chen et al., 2020). Furthermore, due to our more consistent gold frames, we show strong improvements in model generalization in both cross-domain and cross-lingual transfer in low-resource settings. Finally, due to our diminished output vocabulary, we observe 70% reduction in latency and 83% in memory at beam size 5 compared to prior nonautoregressive parsers.

Deliberation Model for On-Device Spoken Language Understanding

“…The final model contains 4.8M parameters and achieves an EM of 83.47% on TOPv2 test. This result is competitive given the model's size, as it is only 3.52% worse than a 134Mparameter RoBERTa-bootstrapped version [23].…”

“…Our pipeline text-based NLU baseline adopts the encoderdecoder non-autoregressive architecture [22] with span pointer network [23]. This setup demonstrated superior results over non-autoregressive and autoregressive alternatives at smaller sizes on TOPv2 [23].…”

“…Our pipeline text-based NLU baseline adopts the encoderdecoder non-autoregressive architecture [22] with span pointer network [23]. This setup demonstrated superior results over non-autoregressive and autoregressive alternatives at smaller sizes on TOPv2 [23]. The architecture consists of a transformer encoder based on the RoBERTa architecture [24]; however we randomly initialize the model instead of bootstrapping it with RoBERTa layers, as using only one RoBERTa layer will already exceed the NLU budget.…”

“…We set the transformer embedding dimension to 256, the number of attention heads to 8, and the number of encoder layers to 3. We follow [22,23] and have a single layer of span pointer transformer decoder. All models adopt the same unigram WordPiece tokenization used by ASR and are trained on TOPv2 train, with training hyperparameters optimized on TOPv2 valid.…”

Improving Top-K Decoding for Non-Autoregressive Semantic Parsing via Intent Conditioning

“…

Semantic parsing (SP) is a core component of modern virtual assistants like Google Assistant and Amazon Alexa. While sequenceto-sequence based auto-regressive (AR) approaches are common for conversational SP, recent studies (Shrivastava et al, 2021) employ non-autoregressive (NAR) decoders and reduce inference latency while maintaining competitive parsing quality. However, a major drawback of NAR decoders is the difficulty of generating top-k (i.e., k-best) outputs with approaches such as beam search.

…”