InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation

Abstract

Assessing the quality of natural language generation (NLG) systems through human annotation is very expensive. Additionally, human annotation campaigns are time-consuming and include non-reusable human labour. In practice, researchers rely on automatic metrics as a proxy of quality. In the last decade, many string-based metrics (e.g., BLEU or ROUGE) have been introduced. However, such metrics usually rely on exact matches and thus, do not robustly handle synonyms. In this paper, we introduce InfoLM a family of untrained metrics that can be viewed as a string-based metric that addresses the aforementioned flaws thanks to a pre-trained masked language model. This family of metrics also makes use of information measures allowing the possibility to adapt InfoLM to different evaluation criteria. Using direct assessment, we demonstrate that InfoLM achieves statistically significant improvement and two figure correlation gains in many configurations compared to existing metrics on both summarization and data2text generation tasks.

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

“…Hashimoto et al (2019) used the same foundation to combine human and automatic evaluation in capturing the trade-off between sampling diverse outputs and achieving the highest possible quality. Pillutla et al (2021) and Colombo et al (2022) expand on these insights and a framework by Djolonga et al (2020) to compare the human-and model-distributions by measuring the extent to which they diverge. A similar approach based on information theory estimates the extent to which a generated summary helps reconstruct the article on which the summary is based (Egan et al, 2022).…”

WikiDes: A Wikipedia-based dataset for generating short descriptions from paragraphs

“…However, they can not compare two strings based on synonyms. InfoLM overcomes this drawback by using a pre-trained masked language model but does not require training to compute similarity scores between summaries and references over the vocabulary by discrete probability distributions [96].…”

Unsupervised Layer-wise Score Aggregation for Textual OOD Detection

“…For future work, we plan to study OOD in sequence labelling tasks (Witon* et al, 2018;Colombo* et al, 2020;Chapuis* et al, 2020a;Colombo et al, 2021a), sequence generation (Colombo* et al, 2019;Jalalzai* et al, 2020;Modi et al, 2020;Colombo et al, 2021e) and fair classification (Colombo et al, 2021d;Pichler et al, 2022) and multimodal scenario (Garcia* et al, 2019;Dinkar* et al, 2020) as well as automatic evaluation (Colombo et al, 2021c;Colombo, 2021a;Staerman et al, 2021b).…”