GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding

Abstract

Grammatical error correction (GEC) is an important NLP task that is currently usually solved with autoregressive sequence-tosequence models. However, approaches of this class are inherently slow due to one-byone token generation, so non-autoregressive alternatives are needed. In this work, we propose a novel non-autoregressive approach to GEC that decouples the architecture into a permutation network that outputs a self-attention weight matrix that can be used in beam search to find the best permutation of input tokens (with auxiliary ins tokens) and a decoder network based on a step-unrolled denoising autoencoder that fills in specific tokens. This allows us to find the token permutation after only one forward pass of the permutation network, avoiding autoregressive constructions. We show that the resulting network improves over previously known non-autoregressive methods for GEC and reaches the level of autoregressive methods that do not use language-specific synthetic data generation methods. Our results are supported by a comprehensive experimental validation on the ConLL-2014 and Write&Improve+LOCNESS datasets and an extensive ablation study that supports our architectural and algorithmic choices.

Exploring Sentence-level Revision Capabilities of LLMs in English for Academic Purposes Writing Assistance

“…Recent advances in GEC also explore non-autoregressive models, which provide faster performance by decoupling the error detection and correction processes, allowing for more dynamic responses to the identified errors [11]. The essence of this task can be succinctly summarized through the following formulation: Given a sentence S err laden with grammatical inaccuracies, the objective of English GEC is to generate a grammatically correct sentence S corr , wherein S corr retains the original intent and content of S err to the greatest extent possible.…”

Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule

“…Gu et al (2019) non-autoregressively refine an output sequence using language-agnostic insertions and deletions. Yakovlev et al (2023) decompose the inference stage into permutation and decoding. First, a permutation network repositions the tokens of an input sequence with possible deletions and inser-tions.…”

Incorporating rich syntax information in Grammatical Error Correction

“…Prior research incorporates detection results as supplementary information for Seq2Seq correction models Yuan et al, 2021b;Li et al, 2023a). The methods proposed by Mallinson et al (2020) and Yakovlev et al (2023) employ the Masked Language Model (MLM) (Devlin et al, 2018) to obtain corrections, which are constrained by mask number. Chen et al (2020) introduces error span detection and correction to address the GEC problem, which allows for flexible corrections while maximizing time efficiency.…”

“…Multi-Encoder (Yuan et al, 2021a) encodes error categories as auxiliary information. GEC-DePend (Yakovlev et al, 2023) with correction by the MLM. TemplateGEC (Li et al, 2023a) uses the output of the GECToR model as supplementary information for Seq2Seq models.…”