DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations

Ghazarian, Sarik; Wen, Nuan; Galstyan, Aram; Peng, Nanyun

doi:10.18653/v1/2022.acl-long.57

Cited by 11 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The pairwise ranking formulation is motivated by previous works on dialogue evaluation (Mesgar et al, 2020;Gao et al, 2020;Zhang et al, 2021a). Compared to direct assessment approaches (Zhang et al, 2021c;Ghazarian et al, 2022), the main advantage of pairwise ranking is that the model can implicitly learn the features that distinguish the good dialogues from the bad ones based on a large quantity of dialogue pairs for a specific quality dimension.…”

Section: General Frameworkmentioning

confidence: 99%

“…all chatbot turns in a dialogue are averaged to derive the single dialogue-level metric score. The second group includes DynaEval (Zhang et al, 2021a) and DEAM (Ghazarian et al, 2022), two stateof-the-art dialogue-level metrics. Detailed metric descriptions are outlined in Appendix C.…”

Section: Baselinesmentioning

confidence: 99%

“…In addition, existing works on model-based dialogue-level metrics Zhang et al, 2021a;Ghazarian et al, 2022;Zhao et al, 2022) focus very much on a single quality dimension. On the contrary, FineD-Eval is capable of multidimensional evaluation and it can provide more fine-grained and interpretable scores.…”

Section: Related Workmentioning

confidence: 99%

“…Both prior works (Zhang et al, 2021a;Yeh et al, 2021) and our experimental results in §5 suggest that such an approach yields sub-optimal dialogue-level evaluation. The reason may be that turn-level metrics do not model the dependency among utterances within multi-turn interactions, it is difficult for them to spot errors that are only obvious after observing the entire conversation (Ghandeharioun et al, 2019;Ghazarian et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

“…There are some metrics that perform multi-turn evaluation. However, they focus only on a single dimension, such as coherence or overall impression (Mesgar et al, 2020;Zhang et al, 2021a;Ghazarian et al, 2022). When evaluating a dialogue, they assign a single score to quantify one aspect of dialogue quality.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Zhang¹,

D’Haro²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent model-based reference-free metrics for open-domain dialogue evaluation exhibit promising correlations with human judgment 1 .However, they either perform turn-level evaluation or look at a single dialogue quality dimension. One would expect a good evaluation metric to assess multiple quality dimensions at the dialogue level. To this end, we are motivated to propose a multi-dimensional dialogue-level metric, which consists of three sub-metrics with each targeting a specific dimension. The sub-metrics are trained with novel self-supervised objectives and exhibit strong correlations with human judgment for their respective dimensions. Moreover, we explore two approaches to combine the submetrics: metric ensemble and multitask learning. Both approaches yield a holistic metric that significantly outperforms individual submetrics. Compared to the existing state-ofthe-art metric, the combined metrics achieve around 16% relative improvement on average across three high-quality dialogue-level evaluation benchmarks.

show abstract

Section: General Frameworkmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Zhang¹,

D’Haro²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Hierarchical information matters! Improving AMR parsing with multi-granularity representation interactions

Sataer,

Fan,

et al. 2024

Information Processing & Management

View full text Add to dashboard Cite

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

Gehrmann

Clark

Sellam³

2023

jair

View full text Add to dashboard Cite

Evaluation practices in natural language generation (NLG) have many known flaws, but improved evaluation approaches are rarely widely adopted. This issue has become more urgent, since neural generation models have improved to the point where their outputs can often no longer be distinguished based on the surface-level features that older metrics rely on. This paper surveys the issues with human and automatic model evaluations and with commonly used datasets in NLG that have been pointed out over the past 20 years. We summarize, categorize, and discuss how researchers have been addressing these issues and what their findings mean for the current state of model evaluations. Building on those insights, we lay out a long-term vision for evaluation research and propose concrete steps for researchers to improve their evaluation processes. Finally, we analyze 66 generation papers from recent NLP conferences in how well they already follow these suggestions and identify which areas require more drastic changes to the status quo.

show abstract

DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations

Cited by 11 publications

References 0 publications

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Hierarchical information matters! Improving AMR parsing with multi-granularity representation interactions

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

Contact Info

Product

Resources

About