Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?

Abstract

Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. In this paper, we introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language, namely, the regularity for performing arbitrary inferences with generalization on composition. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits. A series of experiments show that three neural models systematically draw inferences on unseen combinations of lexical and logical phenomena when the syntactic structures of the sentences are similar between the training and test sets. However, the performance of the models significantly decreases when the structures are slightly changed in the test set while retaining all vocabularies and constituents already appearing in the training set. This indicates that the generalization ability of neural models is limited to cases where the syntactic structures are nearly the same as those in the training set.

Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation

“…Defining disjoint train/test splits is enough to foil truly unsystematic models (e.g., simple look-up tables). However, building on much previous work (Lake and Baroni, 2018;Hupkes et al, 2019;Yanaka et al, 2020;Bahdanau et al, 2018;Goodwin et al, 2020;Geiger et al, 2019), we contend that a randomly constructed disjoint train/test split only diag-noses the most basic level of systematicity. More difficult systematic generalization tasks will only be solved by models exhibiting more complex compositional structures.…”

“…There are often strong intuitions that certain generalization tasks are only solved by models with systematic structures. These tasks are referred to as systematic generalization tasks (Lake and Baroni, 2018;Hupkes et al, 2019;Yanaka et al, 2020;Bahdanau et al, 2018;Geiger et al, 2019;Goodwin et al, 2020).…”

A survey on Recognizing Textual Entailment as an NLP Evaluation

“…The GLUE and SuperGlue datasets include diagnostic sets where annotators manually labeled samples of examples as requiring a broad range of linguistic phenomena. The types of phenomena manu-Proto-Roles (White et al, 2017), Paraphrastic Inference (White et al, 2017, Event Factuality (Poliak et al, 2018b;Staliūnaitė, 2018), Anaphora Resolution (White et al, 2017Poliak et al, 2018b), Lexicosyntactic Inference (Pavlick and Callison-Burch, 2016;Poliak et al, 2018b;Glockner et al, 2018), Compositionality (Dasgupta et al, 2018), Prepositions (Kim et al, 2019), Comparatives (Kim et al, 2019;Richardson et al, 2020), Quantification/Numerical Reasoning (Naik et al, 2018;Kim et al, 2019;Richardson et al, 2020), Spatial Expressions (Kim et al, 2019), Negation (Naik et al, 2018;Kim et al, 2019;Richardson et al, 2020), Tense & Aspect (Kober et al, 2019), Veridicality (Poliak et al, 2018b;, Monotonicity (Yanaka et al, 2019(Yanaka et al, , 2020Richardson et al, 2020), Presupposition (Jeretic et al, 2020), Implicatures (Jeretic et al, 2020), Temporal Reasoning (Vashishtha et al, 2020) ally labeled include lexical semantics, predicateargument structure, logic, and common sense or world knowledge. 14…”

The Extraordinary Failure of Complement Coercion Crowdsourcing

“…In contrast to Dagan et al (2005), the task definitions were short and loose, relying on the annotators' common sense understanding. Many works since have been using the NLI framework and the crowdsourcing procedure associated with it to test models for different language phenomena (Marelli et al, 2014;Lai et al, 2017;Naik et al, 2018;Ross and Pavlick, 2019;Yanaka et al, 2020).…”

“…In light of the low agreements on explicit modeling of the task of complement coercion, we turn to a different crowdsourcing approach which was proven successful for many linguistic phenomena -using NLI as discussed above ( §2). NLI was used to collect data for a wide range of linguistic phenomena: Paraphrase Inference, Anaphora Resolution, Numerical Reasoning, Implicatures and more (White et al, 2017;Poliak et al, 2018;Jeretic et al, 2020;Yanaka et al, 2020;Naik et al, 2018) (see Poliak (2020)). Therefore, we take a similar approach, with similar methodologies, and make use of NLI as an evaluation setup for the complement coercion phenomenon.…”