SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction

Yoshimura, Ryoma; Kaneko, Masahiro; Kajiwara, Tomoyuki; Komachi, Mamoru

doi:10.18653/v1/2020.coling-main.573

Cited by 8 publications

(18 citation statements)

References 19 publications

(15 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reference-based metrics are able to capture this kind of repetition, but reference-less metrics are unable to address it since they use language models that assign better scores to such sentences. This phenomenon has also been reported by Yoshimura et al (2020), who show that reference-based metrics are still quite useful for addressing this problem. We use the Levenshtein distance to address this problem, which is capable of capturing the repetition in this kind of situation and solving this problem of SOME (Yoshimura et al, 2020).…”

Section: Resultssupporting

confidence: 73%

“…This phenomenon has also been reported by Yoshimura et al (2020), who show that reference-based metrics are still quite useful for addressing this problem. We use the Levenshtein distance to address this problem, which is capable of capturing the repetition in this kind of situation and solving this problem of SOME (Yoshimura et al, 2020).…”

Section: Resultssupporting

confidence: 73%

“…Asano et al (2017) used grammaticality, fluency, and meaningpreservation metrics to achieve better correlation with human judgement. Yoshimura et al (2020) did the same for their sub-metrics that are optimized for manual evaluation (SOME). Grammaticality and fluency can be replaced by perplexity scores generated by a language model, such as the Generative Pretrained Transformer 2 (GPT-2) (Radford et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

“…Table 4: The weakness of SOME (Yoshimura et al, 2020) There were two main reasons that pushed us to use a discrete score. First, perplexity scores can vary greatly between sentences.…”

Section: Predictionmentioning

confidence: 99%

See 3 more Smart Citations

Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric

Islam¹,

Magnani²

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

It is difficult to rank and evaluate the performance of grammatical error correction (GEC) systems, as a sentence can be rewritten in numerous correct ways. A number of GEC metrics have been used to evaluate proposed GEC systems; however, each system relies on either a comparison with one or more reference texts-in what is known as the gold standard for reference-based metrics-or a separate annotated dataset to fine-tune the reference-less metric. Reference-based systems have a low correlation with human judgement, cannot capture all the ways in which a sentence can be corrected, and require substantial work to develop a test dataset. We propose a referenceless GEC evaluation system that is strongly correlated with human judgement, solves the issues related to the use of a reference, and does not need another annotated dataset for fine-tuning. The proposed system relies solely on commonly available tools. Additionally, currently available reference-less metrics do not work properly when part of a sentence is repeated as opposed to reference-based metrics. In our proposed system, we look to address issues inherent in reference-less metrics and reference-based metrics.

show abstract

Section: Resultssupporting

confidence: 73%

Section: Resultssupporting

confidence: 73%

Section: Related Workmentioning

confidence: 99%

“…Table 4: The weakness of SOME (Yoshimura et al, 2020) There were two main reasons that pushed us to use a discrete score. First, perplexity scores can vary greatly between sentences.…”

Section: Predictionmentioning

confidence: 99%

See 2 more Smart Citations

Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric

Islam¹,

Magnani²

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…There are a lot of problems that people may not realize . The 28th International Conference on Computational Linguistics (Yoshimura et al 2020)…”

mentioning

confidence: 99%

Optimization of Reference-less Evaluation Metric of Grammatical Error Correction for Manual Evaluations

Yoshimura

Kaneko

Kajiwara

et al. 2021

Journal of Natural Language Processing

Self Cite

View full text Add to dashboard Cite

The development of a reliable automatic evaluation metric of grammatical error correction (GEC) is useful for the research and development of GEC. Since it is difficult to cover all possible reference sentences, previous studies have proposed referenceless metrics. One of them achieved a higher correlation with manual evaluation than reference-based metrics by integrating metrics from the three perspectives of grammaticality, fluency, and meaning preservation. However, the correlation with the manual evaluation can be further improved because they are not considered for optimizing each metric for each manual evaluation. Therefore, in this study, we propose a method of optimizing each metric. Furthermore, we create a dataset with manual evaluation of system output that is ideal for optimization. Experimental results show

show abstract

Revisiting Meta-evaluation for Grammatical Error Correction

Kobayashi,

Mita,

Komachi

2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges, including biases caused by inconsistencies in evaluation granularity and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models, and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.

show abstract

SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction

Cited by 8 publications

References 19 publications

Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric

Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric

Optimization of Reference-less Evaluation Metric of Grammatical Error Correction for Manual Evaluations

Revisiting Meta-evaluation for Grammatical Error Correction

Contact Info

Product

Resources

About