Structural Adapters in Pretrained Language Models for AMR-to-Text Generation

Abstract

Pretrained language models (PLM) have recently advanced graph-to-text generation, where the input graph is linearized into a sequence and fed into the PLM to obtain its representation. However, efficiently encoding the graph structure in PLMs is challenging because such models were pretrained on natural language, and modeling structured data may lead to catastrophic forgetting of distributional knowledge. In this paper, we propose STRUCTADAPT, an adapter method to encode graph structure into PLMs. Contrary to prior work, STRUCTADAPT effectively models interactions among the nodes based on the graph connectivity, only training graph structure-aware adapter parameters. In this way, we incorporate task-specific knowledge while maintaining the topological structure of the graph. We empirically show the benefits of explicitly encoding graph structure into PLMs using STRUCTADAPT, outperforming the state of the art on two AMR-to-text datasets, training only 5.1% of the PLM parameters. 1

Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters

“…Nonetheless, adjacent nodes in the graph may be at multiple positions away from one another in the final serialization. To counteract this, Ribeiro et al (2021b) introduced StructAdapt, a structure-aware (encoder) adapter. It solves the problem of segmented nodes labels by reconstructing a new graph from the resulting subwords.…”

“…GAT is akin to GCN but differs in that aggregation of neighbors embeddings are weighted using an attention mechanism. Unlike GAT and GCN, RGCN further captures the type of the relation between (Ribeiro et al, 2021b). The details of the representations computation for each model can be found in Appendix A.…”

“…A limitation of those is the absence of pretraining, as demonstrated by Ribeiro et al (2021a). To this end, Ribeiro et al (2021b) introduced StructAdapt for lightweight AMR-to-Text with structural adapters. As linearization and tokenization of the input graph are mandatory steps for PLMs, StructAdapt first defines a new graph where nodes are the resulting subwords from the tokenization.…”

Can AMR Assist Legal and Logical Reasoning?

“…We also experiment with adapters since they allow for effective transfer learning (Ribeiro et al, 2021). For this model, the intention is that the adapter parameters can learn the linearised graph representation, while the parameters of the model that hold the distributed knowledge of pre-training are not changed.…”

School Mathematics Textbooks in China

“…While previous work has integrated graphs into neural models for NLP tasks, adding additional neural architectures to PLMs can be non-trivial, as training a graph network without compromising the original architecture of PLMs can be challenging (Ribeiro et al, 2021). Converting AMR graphs directly into text sequences and appending them can be natural, but leads to excessively long sequences, exceeding the maximum process-Figure 1: An example from our experiments.…”