Automatic Generation of Socratic Subquestions for Teaching Math Word Problems

Abstract

Socratic questioning is an educational method that allows students to discover answers to complex problems by asking them a series of thoughtful questions. Generation of didactically sound questions is challenging, requiring understanding of the reasoning process involved in the problem. We hypothesize that such questioning strategy can not only enhance the human performance, but also assist the math word problem (MWP) solvers. In this work, we explore the ability of large language models (LMs) in generating sequential questions for guiding math word problem-solving. We propose various guided question generation schemes based on input conditioning and reinforcement learning. On both automatic and human quality evaluations, we find that LMs constrained with desirable question properties generate superior questions and improve the overall performance of a math word problem solver. We conduct a preliminary user study to examine the potential value of such question generation models in the education domain. Results suggest that the difficulty level of problems plays an important role in determining whether questioning improves or hinders human performance. We discuss the future of using such questioning strategies in education.https://github.com/eth-nlped/ scaffolding-generation

Distilling Reasoning Capabilities into Smaller Language Models

“…Decomposing Multi-Step Reasoning Tasks Solving multi-step reasoning tasks like MWPs has been a popular area of research for the last couple of years Hosseini et al, 2014;Roy et al, 2015;Amini et al, 2019;Zhang et al, 2020;Shridhar et al, 2022;Opedal et al, 2023). However, the majority of the modern approaches for these problems are shifting towards using large language models, often relying on approaches involving prompting or in-context learning (Cobbe et al, 2021;Kojima et al, 2022;Wei et al, 2022b;Chowdhery et al, 2022;Lewkowycz et al, 2022;Srivastava et al, 2022).…”

“…We found that the fine-tuned GPT-2 predicted an incorrect number of subquestions for the majority of problems (see Table 4, first row). Thus, following previous work on subquestion generation (Shridhar et al, 2022), we introduced a guidance mechanism that conditions the generation of subquestions for a problem P on the equations describing the intermediate solutions of P . This strategy improved the quality of the generated questions for all three metrics considered (Table 4, second row).…”

Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses

“…With the emergence of the SQuAD dataset (Rajpurkar et al, 2016), context-dependent QG gained momentum (Du et al, 2017;Yuan et al, 2017;Subramanian et al, 2018;Puri et al, 2020). This extended to complex tasks like generating unanswerable questions (Choi et al, 2018;Zhu et al, 2019;Reddy et al, 2019) and multi-hop reasoning (Pan et al, 2020(Pan et al, , 2021Shridhar et al, 2022). Our work, focusing on generating code tracing questions in CS Education domain, addresses unique challenges around code, natural language, and pedagogical comprehension, inadequately covered by previous methods due to a lack of specialized datasets.…”

The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models

“…More recently, (Patel et al, 2022) proposes an alternative approach to enhance the performance of LLMs by decomposing challenging questions into simpler sub-questions on various tasks. Notably, the efficacy of question decomposition has been demonstrated across a range of tasks and domains, including solving mathematical problems (Shridhar et al, 2022), medical question answering (Roberts et al, 2014), and factual correction (Huang et al, 2023).…”