COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

Abstract

Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts. To facilitate the evaluation of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a fragment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional generalization; these include new combinations of familiar syntactic structures, or new combinations of familiar words and familiar structures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96-99%), but generalization accuracy was substantially lower (16-35%) and showed high sensitivity to random seed (±6-8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress.

Making Transformers Solve Compositional Tasks

“…The main contributions of this paper are: (1) A study of the Transformer architecture design space, showing which design choices result in an inductive learning bias that leads to compositional generalization across a variety of tasks. (2) state-of-the-art results in some of the datasets used, such as COGS, where we report a classification accuracy of 0.784 using an intermediate representation based on sequence tagging (compared to 0.35 for the best previously reported model (Kim and Linzen, 2020)), and the productivity and systematicity splits of PCFG (Hupkes et al, 2020).…”

“…Examples in the structural generalization tasks are typically longer than in the training set and require productivity. All the models tested in the original COGS paper (Kim and Linzen, 2020) (and all of our seq2seq approaches above) achieved 0 accuracy in this category, while performance on lexical tasks is mixed. The small-6s seq2seq model improves the overall performance from 0.278 to 0.475, but curiously has near 0 performance on Verb Argument Structure Alternation tasks, worse than the base abs seq2seq model.…”

“…With the goal of creating general models that generalize compositionally in a large range of tasks, in this paper we explore the design space of Transformer models, showing that several design decisions, such as position encodings, decoder type, weight sharing, model hyper-parameters, and formulation of the target task result in different inductive biases, with significant impact for compositional generalization.In order to evaluate the different design decisions, we use a collection of twelve datasets designed ro measure compositional generalization. In addition to six standard datasets commonly used in the literature (such as SCAN (Lake and Baroni, 2018), PCFG (Hupkes et al, 2020), CFQ (Keysers et al, 2019) and COGS (Kim and Linzen, 2020)), we also use a set of basic algorithmic tasks (such as addition, duplication, or set intersection) that although not directly involving natural language, are useful to obtain insights into what can and cannot be learned with different Transformer models.…”

Are Pre-trained Convolutions Better than Pre-trained Transformers?

“…We show that pre-trained convolutions are competitive against pre-trained Transformers via a set of experiments on a potpourri of NLP tasks, like toxicity detection, sentiment classification, news classification, query understanding and semantic parsing/compositional generalization (Kim and Linzen, 2020). Moreover, we find that pretrained convolutions can outperform, in terms of model quality and training speed, state-of-the-art pre-trained Transformers (Raffel et al, 2019) in certain scenarios.…”

Recursive Decoding: A Situated Cognition Approach to Compositional Generation in Grounded Language Understanding

“…learning system (Chomsky, 2009;Lake, 2014;Lake et al, 2019). However, a range of curated datasets have revealed that standard deep neural networks struggle to generalize compositionally to novel utterances not seen during training (Shridhar et al, 2020;Kim & Linzen, 2020;Lake & Baroni, 2018).…”