Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs

Abstract

We present a novel encoder-decoder architecture for graph-to-text generation based on Transformer, called the Graformer. With our novel graph self-attention, every node in the input graph is taken into account for the encoding of every other node -not only direct neighbors, facilitating the detection of global patterns. For this, the relation between any two nodes is characterized by the length of the shortest path between them, including the special case when there is no such path. The Graformer learns to weigh these node-node relations differently for different attention heads, thus virtually learning differently connected views of the input graph. We evaluate the Graformer on two graph-to-text generation benchmarks, the AGENDA dataset and the WebNLG challenge dataset, where it achieves strong performance while using significantly less parameters than other approaches. 1

A Roadmap for Big Model

“…Recently, graph neural networks (GNNs) have been widely explored for modeling graph structures. Approaches [1379,1380,1381,1382,1383,1384,1385,1386] leverage GNNs and variants to directly encode graph structures. Another line of research [1387,1385,1388,1389] inject the structure information into a sequence-based model, e.g., Transformer.…”

“…Approaches [1379,1380,1381,1382,1383,1384,1385,1386] leverage GNNs and variants to directly encode graph structures. Another line of research [1387,1385,1388,1389] inject the structure information into a sequence-based model, e.g., Transformer. Same as the table-to-text task, pre-training methods also boost the graph-to-text generation.…”

GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation

“…Graph Neural Networks (GNNs) (Veličković et al, 2018) have shown to be effective at encoding graph data. For the KG-to-text task, recent works have leveraged GNNs to encode a graph's neighborhood information (Koncel-Kedziorski et al, 2019;Marcheggiani and Perez-Beltrachini, 2018;Ribeiro et al, 2020b;Schmitt et al, 2021;Guo et al, 2019;Jin et al, 2020) before decoding its corresponding textual representation. Other work instead choose a more global approach and base their encoder on a Transformer-based architecture (Vaswani et al, 2017), calculating self-attention from all the nodes in a graph (Zhu et al, 2019;Cai and Lam, 2020;Ke et al, 2021).…”

Position Information in Transformers: An Overview

“…It is noteworthy, that achieve their best performance by combining both variants of SPR with sequential position information and that SPR as sole sentence representation, i.e., without additional sequential information, leads to a large drop in performance. (Schmitt et al, 2021), showing their definition of relative position encodings in a graph based on the lengths of shortest paths. ∞ means that there is no path between two nodes.…”

“…In contrast to the other approaches, Graformer explicitly models disconnected graphs (∞) and does not add any sequential position information. Unfortunately, Schmitt et al (2021) do not evaluate Graformer on the same tasks as the other discussed approaches, which makes a performance comparison difficult. Decoder: The decoder is also composed of a stack of N = 6 identical layers.…”