Promoting Graph Awareness in Linearized Graph-to-Text Generation

Abstract

Generating text from structured inputs, such as meaning representations or RDF triples, has often involved the use of specialized graphencoding neural networks. However, recent applications of pretrained transformers to linearizations of graph inputs have yielded stateof-the-art generation results on graph-to-text tasks. Here, we explore the ability of these linearized models to encode local graph structures, in particular their invariance to the graph linearization strategy and their ability to reconstruct corrupted inputs. Our findings motivate solutions to enrich the quality of models' implicit graph encodings via scaffolding. Namely, we use graph-denoising objectives implemented in a multi-task text-to-text framework. We find that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.

Structural Adapters in Pretrained Language Models for AMR-to-Text Generation

“…We find that training only 5.1% task-specific parameters, STRUCTADAPT-RGCN achieves a BLEU score of 46.6 in LDC2017T10, substantially improving over FINE-TUNE and other lightweight baselines (ADAPT, FT-TOP2, FT-BOTTOM2), and outperforming Ribeiro et al (2020a) and Hoyle et al (2021) which fine-tune T5 updating significantly more parameters. STRUCTADAPT also achieves stateof-the-art performance on LDC2020T02, considerably improving over Bevilacqua et al (2021), which implicitly models the graph structure information using linearization techniques.…”

Investigating Pretrained Language Models for Graph-to-Text Generation

“…Radev et al (2020) propose DART, a new data-to-text dataset, and train a BART model gradually augmenting the WebNLG training data with DART data. Hoyle et al (2021) explore scaffolding objectives in PLMs and show gains in low-resource graph-to-text settings. Different from the above works, we focus on a general transfer learning strategies for graph-to-text generation, investigating task-adaptive pretraining approaches, employing additional collected task-specific data for different PLMs (BART and T5) and benchmarks.…”

Structural Adapters in Pretrained Language Models for AMR-to-text Generation

“…Ribeiro et al (2020a) investigate encoder-decoder PLMs for graph-to-text generation, and show that adaptive pretraining can lead to notable improvements and that PLMs benefit much more from the graph structure of AMRs than of KGs. Hoyle et al (2020) explore the extent to which PLMs are invariant to graph linearization, finding that models trained on canonical linearizations fail to generalize to meaning-preserving alternatives. Compared to this line of work, which tunes all PLM parameters, our method obtains a further 19x reduction in task-specific parameters, tuning only 5.1% while maintaining comparable performance.…”

“…We thus are interested in measuring the impact of the graph linearization in the models. Following Hoyle et al (2020), we explore three different graph linearizations: (i) CANON: the original order of the canonical human-created linearizations in AMR corpora; (ii) RECONF: the order from the canonical linearization is ignored, except for the top node; 5 and (iii) RANDOM: constructs a linearization from a random node in the graph, disregarding all order information from the canonical format, but it remains a valid traversal of the graph. All linearizations are converted to a sequence of nodes and edges labels using depth-first traversal and used for both training and evaluation.…”