Systematic Generalization: What Is Required and Can It Be Learned?

Abstract

Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated. We compare both types of models in how much they lend themselves to a particular form of systematic generalization. Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them. Our findings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected. We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization. We find that endto-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization. Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

ReaSCAN: Compositional Reasoning in Language Grounding

“…A number of recent approaches involve generating synthetic datasets to evaluate compositional generalization of neural models [10,11,13,28,29,30,31,32]. For instance, [31] proposed CLOSURE, a set of unseen testing splits for the CLEVR dataset [10] which contains synthetically generated natural-looking questions about 3D geometric objects.…”

Compositional properties of emergent languages in deep learning

“…Lake and Baroni (2017) or Loula, Baroni, and Lake (2018)). This happens because neural networks often latch on to dataset-specific regularities instead of distilling syntactic rules in form of logical formulas (Bahdanau et al, 2018).…”

“…To test the generalization properties of different models Bahdanau et al (2018) created the Spatial Queries on Object Pairs (SQOOP) data set, a visual question answering task, where the main challenge is the necessity to generalize to previously unseen combination of known objects and relations. This task is easy for a human to do since we can easily decide whether the statement "there is a golfball under the hat" is true or not even if we have never seen the combination of golf-ball and hat in one sentence before.…”

Ordered Memory

“…Later, Socher et al (2013) shows their effectiveness on sentiment analysis tasks. Recent work has demonstrated that recursive composition of sentences is crucial to systematic generalisation (Bowman et al, 2015;Bahdanau et al, 2018). also demonstrate that architectures like handle syntax-sensitive dependencies better for language-related tasks.…”

“…Despite being successful in language generation tasks, recurrent neural networks (RNNs, Elman (1990)) fail at tasks that explicitly require and test compositional behavior (Lake and Baroni, 2017;Loula et al, 2018). In particular, Bowman et al (2015), and later Bahdanau et al (2018) give evidence that, by exploiting the appropriate compositional structure of the task, models can generalize better to out-of-distribution test examples. Results from Andreas et al (2016) also indicate that recursively composing smaller modules results in better representations.…”