Distilling Multi-Step Reasoning Capabilities of Large Language Models into Smaller Models via Semantic Decompositions

Shridhar, Kumar; Stolfo, Alessandro; Sachan, Mrinmaya

doi:10.48550/arxiv.2212.00193

Cited by 9 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Its computational efficiency makes it practical for real-world situations where cost and speed are important considerations. For instance, our method has the potential to utilize the capabilities of LLMs through knowledge distillation, serving as a student model to improve performance while maintaining efficiency, like (Ho et al, 2022;Magister et al, 2022;Shridhar et al, 2022;Liang et al, 2023a). Therefore, we believe this work has its value to the research field.…”

Section: Discussion On Large Language Modelsmentioning

confidence: 99%

“…In fact, our solver even outperforms the original chain-ofthought approach on the MAWPS dataset. Additionally, our solver has a more transparent structure and is amenable to further fine-tuning and has the potential to combine with LLMs in a knowledge distillation way such as (Ho et al, 2022;Magister et al, 2022;Shridhar et al, 2022;Liang et al, 2023a). Given these strengths, we believe our work represents a meaningful contribution to the field, providing a powerful Seq2Tree model for MWP solving tasks and advancing the research community's understanding of these problems.…”

Section: Large Language Models In Math Wordmentioning

confidence: 99%

See 1 more Smart Citation

Don’t be Blind to Questions: Question-Oriented Math Word Problem Solving

Liang,

Zhang,

Zhang

2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

Solving math word problems (MWP) is a challenging task for natural language processing systems, as it requires to not only identify and comprehend the problem description within the context, but also to deduce a solution in accordance with the posed question. Previous solvers have been found to prioritize the context over the question, resulting in low performance when solving multiple questions under the same context. In this paper, we present a question-oriented strategy to address this issue and improve the generalizability of MWP solvers. Our approach features an entity-aware encoder that enhances the connection between MWP context and question via entities in established dependency graphs, aiming at obtaining better problem representations. Then, a question-guided decoder is trained using a contrastive learning strategy to enhance the question representations. Empirical evaluations on four benchmarks demonstrate that our method outperforms previous solvers and exhibits a favorable balance between efficacy and efficiency in MWP solving. In addition, our solver is not reliant on any specific pre-trained model and demonstrates seamless compatibility with different pre-trained model backbones. Our codes are released at https://github.com/ Zhenwen-NLP/QoS_AACL Context: Mr. Wang rides a bicycle from home to school at 16 kilometers per hour, and can reach the school in 0.2 hours. He walks 4 kilometers per hour. Question 1: How far is Mr. Wang's home from the school? Solutions: MWP-BERT: (16 × 0.2)/4 (Wrong) Ground Truth: 16 × 0.2 Question 2: How many times faster is Mr. Wang walking than cycling? Solutions: MWP-BERT: 16/4 (Wrong) Ground Truth: 4/16 Question 3: How long does it take Mr. Wang to walk to school?

show abstract

Section: Discussion On Large Language Modelsmentioning

confidence: 99%

Section: Large Language Models In Math Wordmentioning

confidence: 99%

Don’t be Blind to Questions: Question-Oriented Math Word Problem Solving

Liang,

Zhang,

Zhang

2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

show abstract

“…(Magister et al, 2022) assesses the efficacy of chain-of-thought explanations in the training of a small model across three disparate tasks, namely arithmetic reasoning, commonsense reasoning, and symbolic reasoning. Furthermore, (Shridhar et al, 2022b) presents Decompositional Distillation, an approach that segments the problem into subproblems to enhance smaller models' performance.…”

Section: Large Language Models For Knowledge Distillation and Data Ge...mentioning

confidence: 99%

“…student models). The majority of prior research has emphasized using the "explanation" component of the CoT approach as the distilled knowledge (Ho et al, 2022;Li et al, 2022a;Shridhar et al, 2022b;Magister et al, 2022). Nonetheless, these methodologies exhibit certain limitations.…”

mentioning

confidence: 99%

Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation

Liang,

Yu,

Rajpurohit

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

“…By doing so, knowledge of the teacher model is effectively distilled into a much smaller student model, allowing a similar level of performance as the teacher at a lower computational cost. Shridhar et al (2022) distill a GPT3 (6B) model into a GPT-2 model for a Chain-of-thought (CoT) reasoning task. Liang et al (2021) propose MixKD to encourage the student model to mimic the teacher's behavior on not only the available training examples but also on interpolated examples.…”

Section: Knowledge Distillationmentioning

confidence: 99%

Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Sahu¹,

Rodríguez²,

Laradji³

et al. 2022

Proceedings of the 4th Workshop on NLP for Conversational AI

View full text Add to dashboard Cite

Data augmentation is a widely employed technique to alleviate the problem of data scarcity. In this work, we propose a prompting-based approach to generate labelled training data for intent classification with off-the-shelf language models (LMs) such as GPT-3. An advantage of this method is that no task-specific LM-fine-tuning for data generation is required; hence the method requires no hyper-parameter tuning and is applicable even when the available training data is very scarce. We evaluate the proposed method in a few-shot setting on four diverse intent classification tasks. We find that GPT-generated data significantly boosts the performance of intent classifiers when intents in consideration are sufficiently distinct from each other. In tasks with semantically close intents, we observe that the generated data is less helpful. Our analysis shows that this is because GPT often generates utterances that belong to a closely-related intent instead of the desired one. We present preliminary evidence that a prompting-based GPT classifier could be helpful in filtering the generated data to enhance its quality. 1

show abstract

Distilling Multi-Step Reasoning Capabilities of Large Language Models into Smaller Models via Semantic Decompositions

Cited by 9 publications

References 0 publications

Don’t be Blind to Questions: Question-Oriented Math Word Problem Solving

Don’t be Blind to Questions: Question-Oriented Math Word Problem Solving

Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation

Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Contact Info

Product

Resources

About