Evaluating the Evaluation of Diversity in Natural Language Generation

Tevet, Guy; Berant, Jonathan

doi:10.18653/v1/2021.eacl-main.25

Cited by 18 publications

(33 citation statements)

References 26 publications

(14 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use sent-BERT as an output diversity metric by using the cosine distance instead of cosine similarity. Our motivation in choosing these diversity metrics is from Tevet and Berant (2020), who identify dist-n and sent-BERT as the best metrics to evaluate two targeted types of diversity-diverse word choice and diverse content, respectively.…”

Section: Discussionmentioning

confidence: 99%

Decoding Methods for Neural Narrative Generation

DeLucia¹,

Mueller²,

Li³

et al. 2021

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

View full text Add to dashboard Cite

Narrative generation is an open-ended NLP task in which a model generates a story given a prompt. The task is similar to neural response generation for chatbots; however, innovations in response generation are often not applied to narrative generation, despite the similarity between these tasks. We aim to bridge this gap by applying and evaluating advances in decoding methods for neural response generation to neural narrative generation. In particular, we employ GPT-2 and perform ablations across nucleus sampling thresholds and diverse decoding hyperparameters-specifically, maximum mutual information-analyzing results over multiple criteria with automatic and human evaluation. We find that (1) nucleus sampling is generally best with thresholds between 0.7 and 0.9; (2) a maximum mutual information objective can improve the quality of generated stories; and (3) established automatic metrics do not correlate well with human judgments of narrative quality on any qualitative metric.

show abstract

Section: Discussionmentioning

confidence: 99%

Decoding Methods for Neural Narrative Generation

DeLucia¹,

Mueller²,

Li³

et al. 2021

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

View full text Add to dashboard Cite

show abstract

“…2) Reference-free metrics directly evaluate the quality of generated texts without references. Since unsupervised metrics like perplexity (Brown et al, 1992) and distinct n-grams (Li et al, 2016) can only provide a task-agnostic result which correlates weakly with human judgments (Hashimoto et al, 2019;Tevet and Berant, 2021), most of the reference-free metrics resort to supervised models. Specifically, they are trained to fit human-annotated ratings / labels (such as discriminator scores (Shen et al, 2017)) or distinguish human-written texts from negative samples (such as UNION (Guan and Huang, 2020)).…”

Section: Evaluation Metric For Text Generationmentioning

confidence: 99%

CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation

Pei¹,

Zhang²,

Lin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. Unsupervised metrics can only provide a task-agnostic evaluation result which correlates weakly with human judgments, whereas supervised ones may overfit task-specific data with poor generalization ability to other datasets. In this paper, we propose an unsupervised reference-free metric called CTRLEval, which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. On top of these tasks, the metric assembles the generation probabilities from a pre-trained language model without any model training. Experimental results show that our metric has higher correlations with human judgments than other baselines, while obtaining better generalization of evaluating generated texts from different models and with different qualities 1 .

show abstract

“…In spite of growing interest in NLG models that produce diverse outputs, there is currently no principled neu-ral method for evaluating the diversity of an NLG system. As described in Tevet and Berant (2021), existing automatic diversity metrics (e.g. Self-BLEU) perform worse than humans on the task of estimating content diversity, indicating a low correlation between metrics and human judgments.…”

Section: Future Directionsmentioning

confidence: 99%

“…An important desideratum of natural language generation (NLG) is to produce outputs that are not only correct but also diverse (Tevet and Berant, 2021). The term "diversity" in NLG is defined as the ability of a generative model to create a set of possible outputs that are each valid given the input and vary as widely as possible in terms of content, language style, and word variability (Gupta et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

Yu¹,

C²,

Qin³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.

show abstract

Evaluating the Evaluation of Diversity in Natural Language Generation

Cited by 18 publications

References 26 publications

Decoding Methods for Neural Narrative Generation

Decoding Methods for Neural Narrative Generation

CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation

Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

Contact Info

Product

Resources

About