Automatic Accuracy Prediction for AMR Parsing

Opitz, Juri; Frank, Anette

doi:10.18653/v1/s19-1024

Cited by 7 publications

(9 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence we assume that GSII's score increments stem from the fact that GSII yields better reconstructions for all systems. In future work, we plan to explore parse quality control (Opitz and Frank, 2019;Opitz, 2020) or ensemble parsing (van Noord and Bos, 2017), to gain more detailed information on the quality of the meaning reconstructions.…”

Section: The Parser: Achilles' Heel Of Mf β ?mentioning

confidence: 99%

Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Opitz

Frank

2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Self Cite

View full text Add to dashboard Cite

Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides wellknown issues from which such metrics suffer, an additional problem arises when applying these metrics for AMR-to-text evaluation, since an abstract meaning representation allows for numerous surface realizations. In this work we aim to alleviate these issues by proposing MF β , a decomposable metric that builds on two pillars. The first is the principle of meaning preservation M: it measures to what extent a given AMR can be reconstructed from the generated sentence using SOTA AMR parsers and applying (finegrained) AMR evaluation metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form F that measures the linguistic quality of the generated text, which we implement using SOTA language models. In two extensive pilot studies we show that fulfillment of both principles offers benefits for AMR-to-text evaluation, including explainability of scores. Since MF β does not necessarily rely on gold AMRs, it may extend to other text generation tasks.

show abstract

Section: The Parser: Achilles' Heel Of Mf β ?mentioning

confidence: 99%

Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Opitz

Frank

2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Self Cite

View full text Add to dashboard Cite

show abstract

“…The inverse task generates text from AMR graphs (Song et al, 2017(Song et al, , 2018Damonte and Cohen, 2019). Opitz and Frank (2019) rate the quality of automatic AMR parses without costly gold data.…”

Section: Related Workmentioning

confidence: 99%

AMR Similarity Metrics from Principles

Opitz

Pârcălăbescu

Frank

2020

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Different metrics have been proposed to compare Abstract Meaning Representation (AMR) graphs. The canonical Smatch metric (Cai and Knight, 2013 ) aligns the variables of two graphs and assesses triple matches. The recent SemBleu metric (Song and Gildea, 2019 ) is based on the machine-translation metric Bleu (Papineni et al., 2002 ) and increases computational efficiency by ablating the variable-alignment. In this paper, i) we establish criteria that enable researchers to perform a principled assessment of metrics comparing meaning representations like AMR; ii) we undertake a thorough analysis of Smatch and SemBleu where we show that the latter exhibits some undesirable properties. For example, it does not conform to the identity of indiscernibles rule and introduces biases that are hard to control; and iii) we propose a novel metric S2 match that is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria. We assess its suitability and show its advantages over Smatch and SemBleu.

show abstract

“…An assessment for the reconstruction quality of single parses would allow researchers to get confidences for the provided scores by MF β or one could conduct the evaluation only on a subset of generations where we are ensured that the quality of the parse reconstruction lies above a certain level. To assess the potential of such a solution, we use a parse quality estimation system (Opitz and Frank, 2019;Opitz, 2020). We then filter all tuples of generated sentences where the estimated quality of the parse lies above 95% F1 score.…”

Section: More Quality Control: Parse Quality Assessmentmentioning

confidence: 99%

Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Opitz¹,

Frank²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Systems that generate sentences from (abstract) meaning representations (AMRs) are typically evaluated using automatic surface matching metrics that compare the generated texts to the texts that were originally given to human annotators to construct AMR meaning representations. However, besides wellknown issues from which such metrics suffer (Callison-Burch et al., 2006;Novikova et al., 2017), we show that an additional problem arises when applied for AMR-to-text evaluation because mapping from the more abstract domain of AMR to the more concrete domain of sentences allows for manifold sentence realizations. In this work we aim to alleviate these issues and propose MF β , an automatic metric that builds on two pillars. The first pillar is the principle of meaning preservation M: it measures to what extent the original AMR graph can be reconstructed from the generated sentence. We implement this principle by i) automatically constructing an AMR from the generated sentence using state-of-the-art AMR parsers and ii) apply fine-grained principled AMR metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form F, which measures the linguistic quality of the generated sentences, which we implement using SOTA language models. We show -theoretically and experimentallythat fulfillment of both principles offers several benefits for evaluation of AMR-to-text systems, including the explainability of scores.

show abstract

Automatic Accuracy Prediction for AMR Parsing

Cited by 7 publications

References 35 publications

Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

AMR Similarity Metrics from Principles

Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Contact Info

Product

Resources

About