Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Rebuffel, Clément; Scialom, Thomas; Soulier, Laure; Piwowarski, Benjamin; Lamprier, Sylvain; Staiano, Jacopo; Scoutheeten, Geoffrey; Gallinari, Patrick

doi:10.18653/v1/2021.emnlp-main.633

Cited by 17 publications

(30 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to the faithfulness classifiers above, these aim to measure whether generated text contains the same information as a source or reference. Instantiations of these metrics may blank out entities (Eyal et al, 2019;Xie et al, 2021;Scialom et al, 2019), or fully generate questions (Chen et al, 2018;Rebuffel et al, 2021;Honovich et al, 2021;Deutsch et al, 2021a, inter alia).…”

Section: The Status Quomentioning

confidence: 99%

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

Gehrmann¹,

Clark²,

Sellam³

2022

Preprint

View full text Add to dashboard Cite

Evaluation practices in natural language generation (NLG) have many known flaws, but improved evaluation approaches are rarely widely adopted.This issue has become more urgent, since neural NLG models have improved to the point where they can often no longer be distinguished based on the surfacelevel features that older metrics rely on. This paper surveys the issues with human and automatic model evaluations and with commonly used datasets in NLG that have been pointed out over the past 20 years. We summarize, categorize, and discuss how researchers have been addressing these issues and what their findings mean for the current state of model evaluations. Building on those insights, we lay out a long-term vision for NLG evaluation and propose concrete steps for researchers to improve their evaluation processes. Finally, we analyze 66 NLG papers from recent NLP conferences in how well they already follow these suggestions and identify which areas require more drastic changes to the status quo.

show abstract

Section: The Status Quomentioning

confidence: 99%

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

Gehrmann¹,

Clark²,

Sellam³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Natural Language Processing One of the sources of QG/QA methods is the thriving field of question generation from natural language processing and information retrieval Jain et al [2018]. Our approach is inspired from text generation methods where QG and QA are used to measure the quality of a generated text without using a human reference , Rebuffel et al [2021].…”

Section: Related Workmentioning

confidence: 99%

“….These two modules fulfil the role of the functions f and h defined in Section 3. EAGER is inspired from works like QuestEval and Data-QuestEval Rebuffel et al [2021], developed for natural language generation. For instance, for abstractive summarization, by generating questions from the original text (QG) and trying to answer them using the summary QA, this method measures the quantity of information shared between both texts.…”

Section: Eagermentioning

confidence: 99%

“…for the evaluation of automatic summary tasks. In that context, some successful approaches have developed reference-less metrics , Rebuffel et al [2021], based on question generation (QG) and question answering (QA). These techniques assess the quality of a generated text by measuring the quantity of information from the source conserved in the generated text (broadly, how well one can answer questions about the original text using the generated text).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL

Carta¹,

Lamprier²,

Oudeyer³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) in long horizon and sparse reward tasks is notoriously difficult and requires a lot of training steps. A standard solution to speed up the process is to leverage additional reward signals, shaping it to better guide the learning process. In the context of language-conditioned RL, the abstraction and generalisation properties of the language input provide opportunities for more efficient ways of shaping the reward. In this paper, we leverage this idea and propose an automated reward shaping method where the agent extracts auxiliary objectives from the general language goal. These auxiliary objectives use a question generation (QG) and question answering (QA) system: they consist of questions leading the agent to try to reconstruct partial information about the global goal using its own trajectory. When it succeeds, it receives an intrinsic reward proportional to its confidence in its answer. This incentivizes the agent to generate trajectories which unambiguously explain various aspects of the general language goal. Our experimental study shows that this approach, which does not require engineer intervention to design the auxiliary objectives, improves sample efficiency by effectively directing exploration.Preprint. Under review.

show abstract

“…Then the matching score is calculated between the answer from the document and the summary. QuestEval (Rebuffel et al, 2021) not only generate (question, answer) pairs from the summary, but also from the source document, which considers to measure the recall performance.…”

Section: Setupmentioning

confidence: 99%

Jointly Learning Guidance Induction and Faithful Summary Generation via Conditional Variational Autoencoders

Wang¹,

Zhao²

2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

ive summarization can generate high quality results with the development of the neural network. However, generating factual consistency summaries is a challenging task for abstractive summarization. Recent studies extract the additional information with off-the-shelf tools from the source document as a clue to guide the summary generation, which shows effectiveness to improve the faithfulness. Unlike these work, we present a novel framework based on conditional variational autoencoders, which induces the guidance information and generates the summary equipped with the guidance synchronously. Experiments on XSUM and CNNDM dataset show that our approach can generate relevant and fluent summaries which is more faithful than the existing state-of-theart approaches, according to multiple factual consistency metrics.

show abstract

Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Cited by 17 publications

References 26 publications

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL

Jointly Learning Guidance Induction and Faithful Summary Generation via Conditional Variational Autoencoders

Contact Info

Product

Resources

About