Posing Fair Generalization Tasks for Natural Language Inference

Geiger, Atticus; Cases, Ignacio; Karttunen, Lauri; Potts, Christopher

doi:10.18653/v1/d19-1456

Cited by 40 publications

(43 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Defining disjoint train/test splits is enough to foil truly unsystematic models (e.g., simple look-up tables). However, building on much previous work (Lake and Baroni, 2018;Hupkes et al, 2019;Yanaka et al, 2020;Bahdanau et al, 2018;Goodwin et al, 2020;Geiger et al, 2019), we contend that a randomly constructed disjoint train/test split only diag-noses the most basic level of systematicity. More difficult systematic generalization tasks will only be solved by models exhibiting more complex compositional structures.…”

Section: A Systematic Generalization Taskmentioning

confidence: 92%

“…There is an extensive literature on monotonicity logics (Moss, 2009;Icard, 2012;Icard and Moss, 2013;. Within NLP, MacCartney andManning (2008, 2009) apply very rich monotonicity algebras to NLI problems, Hu et al (2019a,b) create NLI models that use polarity-marked parse trees, and Yanaka et al (2019a,b) and Geiger et al (2019) investigate the ability of neural models to understand natural logic reasoning. While we consider only a small fragment of these approaches, the methods we develop should apply to more complex systems as well.…”

Section: Related Workmentioning

confidence: 99%

“…There are often strong intuitions that certain generalization tasks are only solved by models with systematic structures. These tasks are referred to as systematic generalization tasks (Lake and Baroni, 2018;Hupkes et al, 2019;Yanaka et al, 2020;Bahdanau et al, 2018;Geiger et al, 2019;Goodwin et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…Previous work evaluating the ability of neural models to learn monotonicity has focused on challenge test sets and systematic generalization tasks (Yanaka et al, 2019b,a;Geiger et al, 2019;Richardson et al, 2019). These behavioral evaluations ask whether models achieve a desired inputoutput behavior.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation

Geiger¹,

Richardson²,

Potts³

2020

Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

Self Cite

View full text Add to dashboard Cite

We address whether neural models for Natural Language Inference (NLI) can learn the compositional interactions between lexical entailment and negation, using four methods: the behavioral evaluation methods of (1) challenge test sets and (2) systematic generalization tasks, and the structural evaluation methods of (3) probes and ( 4) interventions. To facilitate this holistic evaluation, we present Monotonicity NLI (MoNLI), a new naturalistic dataset focused on lexical entailment and negation. In our behavioral evaluations, we find that models trained on general-purpose NLI datasets fail systematically on MoNLI examples containing negation, but that MoNLI fine-tuning addresses this failure. In our structural evaluations, we look for evidence that our top-performing BERT-based model has learned to implement the monotonicity algorithm behind MoNLI. Probes yield evidence consistent with this conclusion, and our intervention experiments bolster this, showing that the causal dynamics of the model mirror the causal dynamics of this algorithm on subsets of MoNLI. This suggests that the BERT model at least partially embeds a theory of lexical entailment and negation at an algorithmic level.

show abstract

Section: A Systematic Generalization Taskmentioning

confidence: 92%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation

Geiger¹,

Richardson²,

Potts³

2020

Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

Self Cite

View full text Add to dashboard Cite

show abstract

“…Salvatore et al (2019) use synthetic data generated from logical forms to evaluate the performance of textual entailment models (e.g., BERT). Geiger et al (2019) use synthetic data to create fair evaluation sets for natural language inference. Geva et al (2020) show the importance of injecting numerical reasoning via generated data into the model to solve reading comprehension tasks.…”

Section: Counterfactual Data Generationmentioning

confidence: 99%

Understanding tables with intermediate pre-training

Eisenschlos¹,

Krichene²,

Müller³

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to finetuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TAB-FACT (Chen et al., 2020) and SQA datasets.

show abstract

Modelling Symbolic Knowledge Using Neural Representations

Schockaert

Gutiérrez-Basulto

2022

Reasoning Web. Declarative Artificial Intelligence

View full text Add to dashboard Cite

Symbolic reasoning and deep learning are two fundamentally different approaches to building AI systems, with complementary strengths and weaknesses. Despite their clear differences, however, the line between these two approaches is increasingly blurry. For instance, the neural language models which are popular in Natural Language Processing are increasingly playing the role of knowledge bases, while neural network learning strategies are being used to learn symbolic knowledge, and to develop strategies for reasoning more flexibly with such knowledge. This blurring of the boundary between symbolic and neural methods offers significant opportunities for developing systems that can combine the flexibility and inductive capabilities of neural networks with the transparency and systematic reasoning abilities of symbolic frameworks. At the same time, there are still many open questions around how such a combination can best be achieved. This paper presents an overview of recent work on the relationship between symbolic knowledge and neural representations, with a focus on the use of neural networks, and vector representations more generally, for encoding knowledge.

show abstract

Posing Fair Generalization Tasks for Natural Language Inference

Cited by 40 publications

References 17 publications

Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation

Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation

Understanding tables with intermediate pre-training

Modelling Symbolic Knowledge Using Neural Representations

Contact Info

Product

Resources

About