Reasoning Like Program Executors

Pi, Xinyu; Liu, Qian; Chen, Bei; Ziyadi, Morteza; Lin, Zeqi; Gao, Yan; Fu, Qiang; Lou, Jian–Guang; Chen, Weizhu

doi:10.48550/arxiv.2201.11473

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The need to prepare students for an increasingly complex world has led to the development of several new training methodologies (e.g., Pi et al, 2022;Wang et al, 2022;Mentzer et al, 2023). Pi and colleagues (2022) provide POET, a pretraining paradigm for basic reasoning that they call simply POET-Math and POET-Logic, as well as the more complex POET-SQL.…”

Section: The Reasoning For Complexity Competencymentioning

confidence: 99%

Identification of Complex Thinking Related Competencies

Talamás-Carvajal,

Ceballos,

Ramírez-Montoya

2024

Learning Analytics

View full text Add to dashboard Cite

Complex thinking competency enhances the high cognitive capacities necessary for the future of education. This study aimed to analyze these capacities through its sub-competencies (critical, systemic, and scientific thinking). We worked with the Cross Industry Standard Process for Data Mining methodology, with an original database of class data of 33,319 unique students, 46 different variables, and a random identification number. The variables were sociodemographic information, academic information, subject admission, competencies, and activities. Statistical analyses identified correlations between competency and sub-competencies. The findings show that 1) critical thinking is strategic in the development of complex thinking and its sub-competencies; 2) Development of Critical Thinking skills early in the curriculum can lead to a cascade effect, enhancing competence and sub-competence development; and 3) an overall performance encompasses the semester results. The study is of value to the academic, technological, and social communities to provide opportunities for the design and implementation of challenging scenarios for the future of education.

show abstract

Section: The Reasoning For Complexity Competencymentioning

confidence: 99%

Identification of Complex Thinking Related Competencies

Talamás-Carvajal,

Ceballos,

Ramírez-Montoya

2024

Learning Analytics

View full text Add to dashboard Cite

show abstract

“…It consists of various sub-skills including commonsense reasoning (Zellers et al, 2018;Talmor et al, 2019;Bhagavatula et al, 2019), numerical reasoning (Dua et al, 2019), arithmetic reasoning (Koncel-Kedziorski et al, 2015;Roy and Roth, 2016;Miao et al, 2020;Cobbe et al, 2021), logical reasoning (Yu et al, 2020), tabular reasoning (Zhu et al, 2021), and so on. Previous efforts in machine learning exploited symbolic systems (Mihaylov and Frank, 2018;Ding et al, 2019;Wang et al, 2022b,a) and pre-training strategies (Deng et al, 2021;Asai and Hajishirzi, 2020;Pi et al, 2022). Recently, large language models with chain-of-thought prompting (Wei et al, 2022b;Wang et al, 2022c;Zhou et al, 2022;Zhang et al, 2022b) demonstrate promising reasoning abilities with appropriately designed prompts, achieving competitive performance on several benchmarks.…”

Section: Reasoning Abilitymentioning

confidence: 99%

Active Prompting with Chain-of-Thought for Large Language Models

Diao¹,

Wang²,

Ye³

et al. 2023

Preprint

View full text Add to dashboard Cite

The increasing scale of large language models (LLMs) brings emergent abilities to various complex tasks requiring reasoning, such as arithmetic and commonsense reasoning. It is known that the effective design of task-specific prompts is critical for LLMs' ability to produce high-quality answers. In particular, an effective approach for complex question-andanswer tasks is example-based prompting with chain-of-thought (CoT) reasoning, which significantly improves the performance of LLMs. However, current CoT methods rely on a fixed set of human-annotated exemplars, which are not necessarily the most effective examples for different tasks. This paper proposes a new method, Active-Prompt, to adapt LLMs to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning). For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful ones to annotate from a pool of taskspecific queries. By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty so as to select the most uncertain questions for annotation. Experimental results demonstrate the superiority of our proposed method, achieving state-ofthe-art on eight complex reasoning tasks. Further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracyuncertainty relationship demonstrate the effectiveness of our method. 1

show abstract

“…We also compare with GenBERT and POET BART (Pi et al, 2022) 9 numbers as reported in the respective papers. GenBERT is a BERT-large model specialized for DROP dataset trained with synthetic data proposed by .…”

Section: Teabreac Improves Model Performancementioning

confidence: 99%

“…Yoran et al ( 2022) created a synthetic dataset using 13 handcrafted multihop QA reasoning patterns applied on wikipedia tables. Lastly, Pi et al (2022) showed that pretraining language models on synthetic dataset derived from input and output of program executors (arithmetic, logic-based and SQL-based) can also improve downstream QA performance. In contrast to these works, we use actual questions from a wide range of real datasets to teach a broad range of multi-hop reasoning skills.…”

Section: Related Workmentioning

confidence: 99%

“…Our use of synthetic contexts to reliably teach broad skills is inspired by three strands of recent RC QA research. One strand has shown that skills learnt over synthetic data can indeed transfer to real datasets Yoran et al, 2022;Pi et al, 2022). A second strand has shown that perturbing the existing (natural) contexts of RC instances in a targeted fashion can reduce artifact-based reasoning (Jia and Liang, 2017;Trivedi et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

Trivedi¹,

Balasubramanian²,

Khot³

et al. 2022

Preprint

View full text Add to dashboard Cite

Question-answering datasets require a broad set of reasoning skills. We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion. Specifically, we use widely available QDMR representations to programmatically create synthetic contexts for real questions in six multihop reasoning datasets. These contexts are carefully designed to avoid common reasoning shortcuts prevalent in real contexts that prevent models from learning the right skills. This results in a pretraining dataset, named TeaBReaC, containing 525K multihop questions (with associated formal programs) covering about 900 reasoning patterns. We show that pretraining standard language models (LMs) on TeaBReaC before fine-tuning them on target datasets improves their performance by up to 13 EM points across 3 multihop QA datasets, with a 30 point gain on more complex questions. The resulting models also demonstrate higher robustness, with a 6-11 point improvement on two contrast sets. Furthermore, TeaBReaC pretraining substantially improves model performance and robustness even when starting with numeracy-aware LMs pretrained using recent methods (e.g., PReasM). Our work thus shows how one can effectively use decomposition-guided contexts to robustly teach multihop reasoning. 1

show abstract

Reasoning Like Program Executors

Cited by 3 publications

References 0 publications

Identification of Complex Thinking Related Competencies

Identification of Complex Thinking Related Competencies

Active Prompting with Chain-of-Thought for Large Language Models

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

Contact Info

Product

Resources

About