Reference-less Quality Estimation of Text Simplification Systems

Martin, Louis; Humeau, Samuel; Mazaré, Pierre-Emmanuel; Bordes, Antoine; Clergerie, Éric Villemonte de la; Sagot, Benoît

doi:10.18653/v1/w18-7005

Cited by 47 publications

(113 citation statements)

References 24 publications

Supporting

Mentioning

108

Contrasting

Unclassified

Order By: Relevance

“…QE leverages both the source sentence and the output simplification to provide additional information on specific behaviours of simplification systems which are not reflected in metrics such as SARI. EASSE uses QE features from Martin et al (2018)'s open-source repository 5 . The QE features currently available are: the compression ratio of the simplification with respect to its source sentence, its Levenshtein similarity, the average number of sentence splits performed by the system, the proportion of exact matches (i.e.…”

Section: Word-level Analysis and Qe Featuresmentioning

confidence: 99%

EASSE: Easier Automatic Sentence Simplification Evaluation

Alva-Manchego¹,

Martin²,

Scarton³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

View full text Add to dashboard Cite

We introduce EASSE, a Python package aiming to facilitate and standardise automatic evaluation and comparison of Sentence Simplification (SS) systems. EASSE provides a single access point to a broad range of evaluation resources: standard automatic metrics for assessing SS outputs (e.g. SARI), wordlevel accuracy scores for certain simplification transformations, reference-independent quality estimation features (e.g. compression ratio), and standard test data for SS evaluation (e.g. TurkCorpus). Finally, EASSE generates easy-to-visualise reports on the various metrics and features above and on how a particular SS output fares against reference simplifications. Through experiments, we show that these functionalities allow for better comparison and understanding of the performance of SS systems.

show abstract

Section: Word-level Analysis and Qe Featuresmentioning

confidence: 99%

EASSE: Easier Automatic Sentence Simplification Evaluation

Alva-Manchego¹,

Martin²,

Scarton³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our goal instead, is to analyze the behavior of the MT system when confronted with ungrammatical input. Reference-less evaluation has also been proposed for text simplification (Martin et al, 2018) and GEC (Napoles et al, 2016), while the grammaticality of MT systems' outputs has been evaluated with target-side contrastive pairs (Sennrich, 2017).…”

Section: Notation and Experimental Settingsmentioning

confidence: 99%

An Analysis of Source-Side Grammatical Errors in NMT

Anastasopoulos¹

2019

Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

The quality of Neural Machine Translation (NMT) has been shown to significantly degrade when confronted with source-side noise. We present the first large-scale study of stateof-the-art English-to-German NMT on real grammatical noise, by evaluating on several Grammar Correction corpora. We present methods for evaluating NMT robustness without true references, and we use them for extensive analysis of the effects that different grammatical errors have on the NMT output. We also introduce a technique for visualizing the divergence distribution caused by a sourceside error, which allows for additional insights.

show abstract

“…Research on Quality Estimation has shown promising results on using reference-less metrics to evaluate generated outputs, allowing the automatic assessment to speed-up and scale. This line of work has started to be applied for simplification (Štajner et al 2016;Martin et al 2018), and we believe it needs to be explored further. In addition, human-based evaluation has been limited to three criteria: grammaticality, meaning preservation, and simplicity.…”

Section: Discussionmentioning

confidence: 99%

Data-Driven Sentence Simplification: Survey and Benchmark

Alva-Manchego

Scarton

Specia

2020

Computational Linguistics

View full text Add to dashboard Cite

Sentence Simplification (SS) aims to modify a sentence in order to make it easier to read and understand. In order to do so, several rewriting transformations can be performed such as replacement, reordering, and splitting. Executing these transformations while keeping sentences grammatical, preserving their main idea, and generating simpler output, is a challenging and still far from solved problem. In this article, we survey research on SS, focusing on approaches that attempt to learn how to simplify using corpora of aligned original-simplified sentence pairs in English, which is the dominant paradigm nowadays. We also include a benchmark of different approaches on common data sets so as to compare them and highlight their strengths and limitations. We expect that this survey will serve as a starting point for researchers interested in the task and help spark new ideas for future developments.

show abstract

Reference-less Quality Estimation of Text Simplification Systems

Cited by 47 publications

References 24 publications

EASSE: Easier Automatic Sentence Simplification Evaluation

EASSE: Easier Automatic Sentence Simplification Evaluation

An Analysis of Source-Side Grammatical Errors in NMT

Data-Driven Sentence Simplification: Survey and Benchmark

Contact Info

Product

Resources

About