Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity

Abstract

Several metrics have been proposed for assessing the similarity of (abstract) meaning representations (AMRs), but little is known about how they relate to human similarity ratings. Moreover, the current metrics have complementary strengths and weaknesses: Some emphasize speed, while others make the alignment of graph structures explicit, at the price of a costly alignment step. In this work we propose new Weisfeiler-Leman AMR similarity metrics that unify the strengths of previous metrics, while mitigating their weaknesses. Specifically, our new metrics are able to match contextualized substructures and induce n:m alignments between their nodes. Furthermore, we introduce a Benchmark for AMR Metrics based on Overt Objectives (Bamboo), the first benchmark to support empirical assessment of graph-based MR similarity metrics. Bamboo maximizes the interpretability of results by defining multiple overt objectives that range from sentence similarity objectives to stress tests that probe a metric’s robustness against meaning-altering and meaning- preserving graph transformations. We show the benefits of Bamboo by profiling previous metrics and our own metrics. Results indicate that our novel metrics may serve as a strong baseline for future work.

Evaluate AMR Graph Similarity via Self-supervised Learning

“…Original datasets contain a set of pairs of sentences with a human-labeled similarity score. Opitz et al (2021) utilized a strong parser to construct AMR graph pairs from sentence pairs and normalized similarity scores to the range [0, 1] to facilitate standardized evaluation. There are 1379 and 4927 test instances in the two datasets, respectively.…”

“…The alignment process, which has been proved as NP-hard, limits the efficiency of SMATCH, especially as the sizes of AMR graphs increase. Another weakness is that the structural matching is insufficient for meaning similarity assessment and fragile in concept synonym replacement and structure deformation (Blloshmi et al, 2020;Opitz et al, 2021).…”

“…WWLK Besides treating AMR graphs as triples and linearized grams, Weisfeiler-Leman AMR similarity metrics (Opitz et al, 2021) consider AMR graphs as high-dimensional objects. They first propagate node embeddings iteratively by incorporating contextualization and then employ Wasserstein Weisfeiler Leman kernel (WWLK) to calculate the minimum cost of transforming one graph to another.…”

FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations

“…In particular, we explore Abstract Meaning Representation (AMR) (Banarescu et al, 2013), a semantic formalism that has received much research interest (Song et al, 2018;Guo et al, 2019;Ribeiro et al, , 2021aOpitz et al, 2020Opitz et al, , 2021Fu et al, 2021) and has been shown to benefit downstream tasks such as spoken language understanding (Damonte et al, 2019), machine translation (Song et al, 2019), commonsense reasoning (Lim et al, 2020), and question answering (Kapanipathi et al, 2021;Bornea et al, 2021).…”

FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations

“…In particular, we explore Abstract Meaning Representation (AMR) (Banarescu et al, 2013), a semantic formalism that has received much research interest (Song et al, 2018;Guo et al, 2019;Ribeiro et al, , 2021aOpitz et al, 2020Opitz et al, , 2021Fu et al, 2021) and has been shown to benefit downstream tasks such as spoken language understanding (Damonte et al, 2019), machine translation (Song et al, 2019), commonsense reasoning (Lim et al, 2020), and question answering (Kapanipathi et al, 2021;Bornea et al, 2021).…”