Parallel Iterative Edit Models for Local Sequence Transduction

Abstract

We present a Parallel Iterative Edit (PIE) model for the problem of local sequence transduction arising in tasks like Grammatical error correction (GEC). Recent approaches are based on the popular encoder-decoder (ED) model for sequence to sequence learning. The ED model auto-regressively captures full dependency among output tokens but is slow due to sequential decoding. The PIE model does parallel decoding, giving up the advantage of modelling full dependency in the output, yet it achieves accuracy competitive with the ED model for four reasons: 1. predicting edits instead of tokens, 2. labeling sequences instead of generating sequences, 3. iteratively refining predictions to capture dependencies, and 4. factorizing logits over edits and their token argument to harness pretrained language models like BERT. Experiments on tasks spanning GEC, OCR correction and spell correction demonstrate that the PIE model is an accurate and significantly faster alternative for local sequence transduction. The code and pre-trained models for GEC are available at https://github. com/awasthiabhijeet/PIE.

FELIX: Flexible Text Editing Through Tagging and Insertion

“…LaserTagger ) is a general approach that has been shown to perform well on a number of text editing tasks, but it has two limitations: it does not allow for arbitrary reordering of the input tokens; and insertions are restricted to a fixed phrase vocabulary that is derived from the training data. Similarly, Ed-itNTS (Dong et al, 2019) and PIE (Awasthi et al, 2019) are two other text-editing models developed specifically for simplification and grammatical error correction, respectively.…”

Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection

“…Choe improved the model performance by dividing the GEC task into the generation mechanism, which produces new words, and the copy mechanism, which copies the words of the entered sentence and trains them separately [6]. Awasthi limited the parallel iterative edit [1] and applied BERT [10]. Chollampatt conducted research that applied the CNN model and quality estimation to the GEC task [8].…”

“…The difference with GLEU is that it considers the source sentence and it is a performance evaluation metric that is specialized for the correction system. The majority of current research uses this metric as the official metric of GEC [1,6,7,8,13,16,17,27].…”

“…However, the majority of recent studies have considered only the first factor. With the emergence of deep learning, the GEC task is being considered from the viewpoint of Machine Transla-tion(MT) [1,6,7,8,13,16,17,27]. GEC is regarded as a task that translates a sentence with errors into a correct sentence.…”

“…• The performance measurement was applied in a test that was constructed using a commercialized system to reflect the real world. Moreover, the model developed in this paper was officially distributed in the form of a platform 1 . The remainder of this paper is organized as follows: Section 2 reviews the related work in the area of NMT-based GEC and provides background for improved understanding of our proposed metric.…”