DeepA2: A Modular Framework for Deep Argument Analysis with Pretrained Neural Text2Text Language Models

Betz, Gregor; Richardson, Kyle

doi:10.18653/v1/2022.starsem-1.2

Cited by 6 publications

(5 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Logical reasoning with LLMs and artificially controlled experiments: Integrating logical reasoning ability into neural models is a pivotal goal in the artificial intelligence field (Marcus, 2003). With this aim, enclosing the models' exact weakness with artificially controlled data has been actively conducted in our field (Betz et al, 2021;Clark et al, 2020;Lu et al, 2021;Kudo et al, 2023); we show the peculiar case that just the flip of one word (adding a nation prefix) causes drastic effects for modern LLMs.…”

Section: Related Workmentioning

confidence: 82%

Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism

Ye,

Kuribayashi,

Suzuki

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large language models (LLMs) take advantage of step-by-step reasoning instructions, e.g., chain-of-thought (CoT) prompting. Building on this, their ability to perform CoT-style reasoning robustly is of interest from a probing perspective. In this study, we inspect the stepby-step reasoning ability of LLMs with a focus on negation, which is a core linguistic phenomenon that is difficult to process. In particular, we introduce several controlled settings (e.g., reasoning on fictional entities) to evaluate the logical reasoning abilities of the models. We observed that dozens of modern LLMs were not robust against lexical negation (e.g., plausi-ble→implausible) when performing CoT-style reasoning, and the results highlight unique limitations in each LLM family. https://github.com/muyo8692/ stepbystep-reasoning-vs-negation 14753 Setting Few-shot exemplars Target example If fails at this setting BASE Is a sentence "A does B" plausible? A is a C player. B happens in C/X. So the answer is yes/no. Is a sentence "D does E" plausible? D is a F player. E happens in F/Y. So the answer is __ CoT-style reasoning fails. FIC Is a sentence "A does B" plausible? A is a C player. B happens in C/X. So the answer is yes/no. Is a sentence "α does β" plausible? α is a γ player. β happens in γ/χ. So the answer is __ Reasoning cannot be abstracted to fictional texts. FICNEG Is a sentence "A does B" implausible? A is a C player. B happens in C/X. So the answer is yes/no.Is a sentence "α does β" implausible? α is a γ player. β happens in γ/χ.

show abstract

Section: Related Workmentioning

confidence: 82%

Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism

Ye,

Kuribayashi,

Suzuki

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Automated reasoning has been a challenging task in NLP. Before the era of LLMs, the prevalent approaches to logical reasoning were based on fine-tuning pre-trained models (Clark, Tafjord, and Richardson 2020;Betz, Voigt, and Richardson 2021;Han et al 2022). However, these methods often led to unrealistic inferences due to implicit label-data correlations (Zhang et al 2023).…”

Section: Reasoning With Llmsmentioning

confidence: 99%

MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models

Wu,

Han,

Song

et al. 2024

AAAI

View full text Add to dashboard Cite

Large language models (LLMs) have demonstrated remarkable performance in various natural language processing tasks. However, they still face significant challenges in automated reasoning, particularly in scenarios involving multi-step reasoning. In this paper, we focus on the logical reasoning problem. The main task is to answer a question based on a set of available facts and rules. A lot of work has focused on guiding LLMs to think logically by generating reasoning paths, ignoring the structure among available facts. In this paper, we propose a simple approach MindMap by introducing evidence chains for supporting reasoning. An evidence chain refers to a set of facts that involve the same subject. In this way, we can organize related facts together to avoid missing important information. MindMap can be integrated with existing reasoning framework, such as Chain-of-Thought (CoT) and Selection-Inference (SI), by letting the model select relevant evidence chains instead of independent facts. The experimental results on the bAbI and ProofWriterOWA datasets demonstrate the effectiveness of MindMap.It can significantly improve CoT and SI, especially in multi-step reasoning tasks.

show abstract

“…Another straightforward approach for text-based logical reasoning is to first translate natural language statements into formal logic expressions and then use a formal logic inference engine. A lot of efforts have been made in this direction (Weber et al, 2019;Levkovskyi & Li, 2021;Lu et al, 2022a;Betz & Richardson, 2022), and we also tried it in our experiments. However, it turns out to be very challenging to map natural language to formal logic: the translated formal logic expressions often bear subtle issues (e.g., naming variation) such that a formal logic engine won't be able to pattern-match them well.…”

Section: Related Workmentioning

confidence: 99%

Explicit Planning Helps Language Models in Logical Reasoning

Zhao¹,

Wang²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Language models have been shown to perform remarkably well on a wide range of natural language processing tasks. In this paper, we propose a novel system that uses language models to perform multi-step logical reasoning. Our system incorporates explicit planning into its inference procedure, thus able to make more informed reasoning decisions at each step by looking ahead into their future effects. In our experiments, our full system significantly outperforms other competing systems. On a multiple-choice question answering task, our system performs competitively compared to GPT-3-davinci despite having only around 1.5B parameters. We conduct several ablation studies to demonstrate that explicit planning plays a crucial role in the system's performance.

show abstract

DeepA2: A Modular Framework for Deep Argument Analysis with Pretrained Neural Text2Text Language Models

Cited by 6 publications

References 25 publications

Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism

Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism

MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models

Explicit Planning Helps Language Models in Logical Reasoning

Contact Info

Product

Resources

About