The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Csordás, Róbert; Irie, Kazuki; Schmidhuber, Jürgen

doi:10.18653/v1/2021.emnlp-main.49

Cited by 31 publications

(58 citation statements)

References 18 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Improving generalization Many approaches have been recently proposed to examine and improve generalization, including the effect of training size and architecture , data augmentation (Andreas, 2020;Akyürek et al, 2021;Guo et al, 2021), data sampling (Oren et al, 2021), model architecture Bogin et al, 2021b;Chen et al, 2020), intermediate representations and different training techniques (Oren et al, 2020;Csordás et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Unobserved Local Structures Make Compositional Generalization Hard

Bogin¹,

Gupta²,

Berant³

2022

Preprint

View full text Add to dashboard Cite

While recent work has convincingly showed that sequence-to-sequence models struggle to generalize to new compositions (termed compositional generalization), little is known on what makes compositional generalization hard on a particular test instance. In this work, we investigate what are the factors that make generalization to certain test instances challenging. We first substantiate that indeed some examples are more difficult than others by showing that different models consistently fail or succeed on the same test instances. Then, we propose a criterion for the difficulty of an example: a test instance is hard if it contains a local structure that was not observed at training time. We formulate a simple decision rule based on this criterion and empirically show it predicts instance-level generalization well across 5 different semantic parsing datasets, substantially better than alternative decision rules. Last, we show local structures can be leveraged for creating difficult adversarial compositional splits and also to improve compositional generalization under limited training budgets by strategically selecting examples for the training set.

show abstract

Section: Related Workmentioning

confidence: 99%

Unobserved Local Structures Make Compositional Generalization Hard

Bogin¹,

Gupta²,

Berant³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Improving Generalization Many approaches to examining and improving compositional generalization have been proposed, including specialized architectures with inductive bias for compositional generalization Bogin et al, 2021b;Gordon et al, 2020;, data augmentation (Andreas, 2020; Akyürek et al, 2021;Guo et al, 2021), modifications to training methodology (Oren et al, 2020;Csordás et al, 2021) and meta learning (Conklin et al, 2021;Lake, 2019). Data-based approaches have the advantage of being modelagnostic and hence can be used in conjunction with pretrained models.…”

Section: Related Workmentioning

confidence: 99%

Structurally Diverse Sampling Reduces Spurious Correlations in Semantic Parsing Datasets

Gupta¹,

Singh²,

Gardner³

2022

Preprint

View full text Add to dashboard Cite

A rapidly growing body of research has demonstrated the inability of NLP models to generalize compositionally and has tried to alleviate it through specialized architectures, training schemes, and data augmentation, among other approaches. In this work, we study a different relatively under-explored approach: sampling diverse train sets that encourage compositional generalization. We propose a novel algorithm for sampling a structurally diverse set of instances from a labeled instance pool with structured outputs. Evaluating on 5 semantic parsing datasets of varying complexity, we show that our algorithm performs competitively with or better than prior algorithms in not only compositional template splits but also traditional IID splits of all but the least structurally diverse datasets. In general, we find that diverse train sets lead to better generalization than random training sets of same size in 9 out of 10 dataset-split pairs, with over 10% absolute improvement in 5, providing further evidence to their sample efficiency. Moreover, we show that structural diversity also makes for more comprehensive test sets that require diverse training to succeed on. Finally, we use information theory to show that reduction in spurious correlations between substructures may be one reason why diverse training sets improve generalization.

show abstract

“…Early self-attention mechanisms have added representations of the absolute positions of tokens to its inputs (Vaswani et al, 2017). However, we use representations of relative positions, or distances between tokens, in line with recent work showing that relative attention is advantageous, particularly on length-generalization tasks (Shaw et al, 2018;Csordás et al, 2021). By considering logarithmic distances, our model is also encouraged to attend to more recent tokens during decoding, which can be desirable when programs consist of multiple smaller parts.…”

Section: Baseline Transformermentioning

confidence: 99%

Compositional Generalization and Decomposition in Neural Program Synthesis

Shi¹,

Hong²,

Zaheer³

et al. 2022

Preprint

View full text Add to dashboard Cite

When writing programs, people have the ability to tackle a new complex task by decomposing it into smaller and more familiar subtasks. While it is difficult to measure whether neural program synthesis methods have similar capabilities, what we can measure is whether they compositionally generalize, that is, whether a model that has been trained on the simpler subtasks is subsequently able to solve more complex tasks. In this paper, we focus on measuring the ability of learned program synthesizers to compositionally generalize. We first characterize several different axes along which program synthesis methods would be desired to generalize, e.g., length generalization, or the ability to combine known subroutines in new ways that do not occur in the training data. Based on this characterization, we introduce a benchmark suite of tasks to assess these abilities based on two popular existing datasets, SCAN and RobustFill. Finally, we make first attempts to improve the compositional generalization ability of Transformer models along these axes through novel attention mechanisms that draw inspiration from a humanlike decomposition strategy. Empirically, we find our modified Transformer models generally perform better than natural baselines, but the tasks remain challenging.

show abstract

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Cited by 31 publications

References 18 publications

Unobserved Local Structures Make Compositional Generalization Hard

Unobserved Local Structures Make Compositional Generalization Hard

Structurally Diverse Sampling Reduces Spurious Correlations in Semantic Parsing Datasets

Compositional Generalization and Decomposition in Neural Program Synthesis

Contact Info

Product

Resources

About