Thinking Like a Skeptic: Defeasible Inference in Natural Language

Abstract

Defeasible inference is a mode of reasoning in which an inference (X is a bird, therefore X flies) may be weakened or overturned in light of new evidence (X is a penguin). Though long recognized in classical AI and philosophy, defeasible inference has not been extensively studied in the context of contemporary data-driven research on natural language inference and commonsense reasoning. We introduce Defeasible NLI (abbreviated δ-NLI), a dataset for defeasible inference in natural language. δ-NLI contains extensions to three existing inference datasets covering diverse modes of reasoning: common sense, natural language inference, and social norms. From δ-NLI, we develop both a classification and generation task for defeasible inference, and demonstrate that the generation task is much more challenging. Despite lagging human performance, however, generative models trained on this data are capable of writing sentences that weaken or strengthen a specified inference up to 68% of the time.

Could you give me a hint ? Generating inference graphs for defeasible reasoning

“…In order to establish comparable gains, we replicate the evaluation setup of Rudinger et al (2020) by using use the same Amazon Mechanical Turk template and the instruction set, and the same pool of 230 qualified annotators that Rudinger et al (2020) selected based on a paid qualification test, in which the workers were asked to answer SNLI queries of varying levels of difficulty. We paid slightly above $15 per hour for the tasks.…”

“…Next, we show two examples (Figure 7, Figure 8) where humans were previously unsuccessful on this answer (in the original setup of (Rudinger et al, 2020)), and were successful now having looked at the inference graphs. The humans marked that the mediator nodes and the contextualizer nodes provide useful information.…”

“…Despite their prominence (Bentahar et al, 2010), argumentative models are not scalable because an inference graph needs to be handcrafted for every example. Recently, Rudinger et al (2020) proposed two auxiliary tasks related to defeasible inference: (i) an NLI task to predict whether an update U would weaken or strengthen a hypothesis H, and (ii) a generative task to generate an update U given a premise P and a hypothesis H. However, this only addresses a part of the problem because their inference is still not supported by the line of reasoning that a human typically uses to solve this task, namely mediators (e.g., hallucinations can be deceptive) and contextualizers (some elephants can have mutated gene which makes them look different) that are inherently embedded in an inference graph, limiting their utility for humans (figure 1).…”

Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?

“…A range of recent works are based on the general idea of models revising their behavior according to changes in their input (Wallace et al, 2019;Emelin et al, 2020;Ye and Ren, 2021;Schick and Schütze, 2020;Sheng et al, 2020). For example, Rudinger et al (2020) explore a model's ability to alter its confidence upon observing new facts. show that models can take in rules and perform soft reasoning on them.…”

Modeling Event Plausibility with Consistent Conceptual Abstraction

“…Non-monotonic inferences have recently been explored in the context of defeasible reasoning (Rudinger et al, 2020): inferences that may be strengthened or weakened given additional evidence. The change in plausibility between an event and its abstraction can be formulated as a type of defeasible inference, and our findings may contribute to future work in this area.…”