The price of debiasing automatic metrics in natural language evalaution

Abstract

For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements. On the other hand, averaging human judgments, the unbiased gold standard, is often too expensive. In this paper, we use control variates to combine automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than human evaluation alone. In practice, however, we obtain only a 7-13% cost reduction on evaluating summarization and open-response question answering systems. We then prove that our estimator is optimal: there is no unbiased estimator with lower cost. Our theory further highlights the two fundamental bottlenecks-the automatic metric and the prompt shown to human evaluatorsboth of which need to be improved to obtain greater cost savings. arXiv:1807.02202v1 [cs.CL] 6 Jul 2018Question and reference answer System answer (System; Corr / ROUGE-L)Examples where system is correct and ROUGE-L > 0.5 (19.6% or 285 of 1455 unique responses) Q. what is anti-mullerian hormone A. Anti-Mullerian Hormone (AMH) is a protein hormone produced by granulosa cells (cells lining the egg sacs or follicles) within the ovary.it is a protein hormone produced by granulosa cells (cells lining the egg sacs or follicles) within the ovary. (snet.ens; / 0.86) Examples where system is incorrect and ROUGE-L > 0.5 (1.3% or 19 of 1455 unique responses) Q. at what gestational age can you feel a fetus move A. 37 to 41 weeks (incorrect reference answer) 37 to 41 weeks (fastqa, fastqa.ext; × / 1.0) Examples where system is correct and ROUGE-L < 0.5 (56.0% or 815 of 1455 unique responses) Q. what is the definition of onomatopoeia A. It is defined as a word, which imitates the natural sounds of a thing. the naming of a thing or action by a vocal imitation of the sound associated with it (as buzz, hiss). (fastqa; / 0.23) Examples where system is incorrect and ROUGE-L < 0.5 (23.1% or 336 of 1455 unique responses) Q. what kind root stem does a dandelion have A. Fibrous roots and hollow stem. vitamin a, vitamin c, vitamin d and vitamin b complex, as well as zinc, iron and potassium. (snet, snet.ens; × / 0.09) (a) MS MARCO. Human annotators rated answer correctness (AnyCorrect) and the automatic metric used is ROUGE-L (higher is better).Reference summary System summary (System; Edit / VecSim) Examples where system Edit < 0.3 and VecSim > 0.5 (53.9% or 1078 of 2000 responses)Bhullar is set to sign a -day contract with the Kings.The -year-old will become the NBA's first player of Indian descent. Bhullar will be on the roster when the Kings host New Orleans Pelicans.Bhullar andThe Kings are signing Bhullar to a -day contract.The -year-old will be on the roster on friday when David Wear's -season contract expires thursday. Bhullar is set to become the NBA's first player of Indian descent. (ml; 0.13 / 0.82)Examples where system Edit > 0.3 and VecSim > 0.5 (18.0% or 360 of 2000 responses)

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

“…Results reveal that the lexical metric SENTBLEU can correctly assign lower scores to system translations of low quality, while it struggles in judging system translations of high quality by assigning them lower scores. Our finding agrees with the observations found in Chaganty et al (2018); Novikova et al (2017): lexical metrics correlate better with human judgments on texts of low quality than high quality. Peyrard (2019b) further show that lexical metrics cannot be trusted because Machine Translation (zh-en) Figure 3: Correlation in similar language (de-en) and distant language (zh-en) pair, where bordered area shows correlations between human assessment and metrics, the rest shows inter-correlations across metrics and DA is direct assessment rated by language experts.…”

Extractive NarrativeQA with Heuristic Pre-Training

“…METEOR, in contrast takes synonymy into account, and our methods outperformed previous systems in this metric. Our observation follows recent published work on evaluating abstractive NLI systems (Chaganty et al, 2018). Concurrently with improving NLI methodology, it is worth investing in the development of evaluation methods that reflect progress faithfully.…”

“…We also find that automatic evaluation scores like BLEU and METEOR, which rely on word overlap, are overly conservative regarding the output of our model. A series of recent papers discussed problems of comparing models on abstractive NLI tasks using automatic metrics as the ones listed above (Novikova et al, 2017;Chaganty et al, 2018). While there is decent agreement between human and automatic judgments on bad model outputs, disagreements tend to be substantial on good outputs.…”

Handling Divergent Reference Texts when Evaluating Table-to-Text Generation

“…Sentence Level Discrimination. Chaganty et al (2018) point out that hill-climbing on an automatic metric is meaningless if that metric has a low instance-level correlation to human judgments. In Table 3 we show the average accuracy of the metrics in making the same judgments as humans between pairs of generated texts.…”