Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU

Abstract

While humans process language incrementally, the best language encoders currently used in NLP do not. Both bidirectional LSTMs and Transformers assume that the sequence that is to be encoded is available in full, to be processed either forwards and backwards (BiL-STMs) or as a whole (Transformers). We investigate how they behave under incremental interfaces, when partial output must be provided based on partial input seen up to a certain time step, which may happen in interactive systems. We test five models on various NLU datasets and compare their performance using three incremental evaluation metrics. The results support the possibility of using bidirectional encoders in incremental mode while retaining most of their non-incremental quality. The "omni-directional" BERT model, which achieves better non-incremental performance, is impacted more by the incremental access. This can be alleviated by adapting the training regime (truncated training), or the testing procedure, by delaying the output until some right context is available or by incorporating hypothetical right contexts generated by a language model like GPT-2.

Best of Both Worlds: Making High Accuracy Non-incremental Transformer-based Disfluency Detection Incremental

“…The recent live incremental systems fall short of the same accuracies achievable on pre-segmented transcripts, so there is a natural interest in using the best non-incremental sequence models and adapting them for incrementality. Madureira and Schlangen (2020) take up this effort in several other sequence tagging and classification tasks, showing how bidirectional encoders and Transformers can be modified to work incrementally. To reduce the impact of the partiality of the input, the models predict future content and wait for more rightward context.…”

“…Prophecy-based decoding For our other decoding strategies, we use a 'prophecy'-based approach to predicting future word sequences, following the task of open-ended language generation, which, given an input text passage as context, is to produce text that constitutes a cohesive continuation (Holtzman et al, 2019). Inspired by (Madureira and Schlangen, 2020), using the GPT-2 language model (Radford et al, 2019), we first give each word as a left context and create a continuation until the end of an utterance to create a hypothetical complete context that satisfies the requirements of the models' non-incremental structure.…”

Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU

“…Zhang et al (2021) introduced an average embedding layer to avoid recalculation when using an incremental encoder, while exploiting right context through knowledge distillation. An investigation of the use of non-incremental encoders for incremental NLU in interactive systems was conducted by Madureira and Schlangen (2020). The authors employed BERT (Devlin et al, 2019) for sequence tagging and classification using restart-incrementality, a procedure with high computational cost.…”

“…LT+R+CM+D: similar to (4), but, during training, the output for the input token x t is obtained at time t + d, where d ∈ {1, 2} is the delay, following the approach in Turek et al (2020). There is evidence that additional right context improve the models' incremental performance (Baumann et al, 2011;Ma et al, 2019;Madureira and Schlangen, 2020), which results in a trade-off between providing timely output or waiting for more context to deliver more stable output.…”

“…One way to employ non-incremental models in incremental settings is resorting to an incremental interface, like in Beuck et al (2011), where a complete recomputation of the available partial input happens at each time step to deliver partial output. Madureira and Schlangen (2020) examined the output stability of non-incremental encoders in this restart-incremental fashion. While qualitatively feasible, this procedure is computationally costly, especially for long sequences, since it requires as many forward passes as the number of input tokens.…”