Encode, Tag, Realize: High-Precision Text Editing

Abstract

We propose LASERTAGGER-a sequence tagging approach that casts text generation as a text editing task. Target texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. To predict the edit operations, we propose a novel model, which combines a BERT encoder with an autoregressive Transformer decoder. This approach is evaluated on English text on four tasks: sentence fusion, sentence splitting, abstractive summarization, and grammar correction. LASERTAGGER achieves new state-ofthe-art results on three of these tasks, performs comparably to a set of strong seq2seq baselines with a large number of training examples, and outperforms them when the number of examples is limited. Furthermore, we show that at inference time tagging can be more than two orders of magnitude faster than comparable seq2seq models, making it more attractive for running in a live environment.

Heterogeneous Recycle Generation for Chinese Grammatical Error Correction

“…Currently, the only sequence editing model to be applied to GEC is LaserTagger (Malmi et al, 2019). Similarly to the two previously cited works, LaserTagger learns to edit sentences by two different edit operations: KEEP and DELETE, along with pairing these operations with a limited phrase vocabulary consisting of tokens that are frequently changed between the source and target sequences.…”

“…As described in Malmi et al (2019), a sequence editing model learns to generate a target sentence by applying a small set of edit operations to the source sentence. It works in three steps: (1) the input sentence is encoded into a hidden representation, (2) each token in the input sentence is assigned an edit tag, and (3) rules are applied to convert the output tags into tokens.…”

Sound Natural: Content Rephrasing in Dialog Systems

“…Sentence simplification (Nisioi et al, 2017) aims at using techniques such as shortening the sentences to make a text more readable. On the other hand, style transfer is the task of making an utterance conform to a specific style such as formality (Logeswaran et al, 2018;Sennrich et al, 2016).…”

Recurrent Inference in Text Editing

“…Tagging solves text editing in two steps instead. It firstly employs a seq2seq framework to produce tag sequences, and secondly, edits input texts according to the tag sequences (the "realization" step) (Malmi et al, 2019). Tagging assigns the tag KEEP for words that do not need to be changed so that it does not need to learn a copy mechanism.…”

“…Upon receiving the encoder's hidden states that comprise the source text information, the decoder of End2end directly decodes the hidden states and generates the completely edited target text sequence. But, the decoder of Tagging produces a sequence of editing operations, such as deletion and insertion, that is later applied to the source text to yield the edited text via a realization step (Malmi et al, 2019). The mechanisms of End2end and Tagging are illustrated in Figure 1.…”