Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses

Abstract

Evaluation of grammatical error correction (GEC) systems has primarily focused on essays written by non-native learners of English, which however is only part of the full spectrum of GEC applications. We aim to broaden the target domain of GEC and release CWEB, a new benchmark for GEC consisting of website text generated by English speakers of varying levels of proficiency. Website data is a common and important domain that contains far fewer grammatical errors than learner essays, which we show presents a challenge to stateof-the-art GEC systems. We demonstrate that a factor behind this is the inability of systems to rely on a strong internal language model in low error density domains. We hope this work shall facilitate the development of opendomain GEC models that generalize to different topics and genres.

TGEA: An Error-Annotated Dataset and Benchmark Tasks for TextGeneration from Pretrained Language Models

“…It uses articles or blogs (e.g., Wiki, Yahoo)) written by native English speakers to explore grammatical error phenomena in different domains. CWEB (Flachs et al, 2020) also uses website texts in English, such as blogs. The difference between CWEB and CMEG is that the percentage of erroneous tokens in the former is smaller than the latter as the purpose of CWEB is to study grammatical error correction in low error density domains.…”

“…Such errors include not only lexical collocation errors but also longdistance syntactic constituency combination errors (e.g., inappropriate subject-object combination). This error type is similar to "replacing" error in some GEC datasets (e.g., CWEB (Flachs et al, 2020)) as one element of an inappropriate combination should be usually replaced with other expressions. As we want to find text spans associated with erroneous words/phrases, we term this error type as "inappropriate combination".…”

“…For erroneous span detection, as a text may contain several spans that can be edited or the text can be corrected in different ways, which span should be regarded as erroneous is closely related to the way that we correct the text. Therefore, the basic principle that guides the annotation of erro-neous spans is also the rule that we use for error correction: making minimal edits, which is also used in GEC datasets (Flachs et al, 2020;Napoles et al, 2017). In addition to the minimal edit principle, we also provide the following specific rules for annotators:…”

“…First, comparing with GEC data on human-written texts, our dataset is relatively small. Second, our dataset contains error types that are very different from those in previous GEC datasets (Zhao et al, 2018;Flachs et al, 2020). Punctuation, spelling and other word-characterlevel errors, which are easy to be corrected, are rare in TGEA although they are quite common in GEC datasets.…”

“…Therefore it is different from conventional text generation datasets (e.g., Multi-News (Fabbri et al, 2019), TextCaps (Sidorov et al, 2020)) that are constructed to train models to learn text generation (e.g., generating texts from images or long documents). It is also different from grammatical error correction (GEC) datasets (Zhao et al, 2018;Flachs et al, 2020) that are built from human-written texts usually by second language learners. • TGEA provides rich semantic information for text generation errors, including error types, associated text spans, error corrections and rationals behind errors, as shown in Figure 1.…”