Improving Math Word Problems with Pre-trained Knowledge and Hierarchical Reasoning

Abstract

The recent algorithms for math word problems (MWP) neglect to use outside knowledge not present in the problems. Most of them only capture the word-level relationship and ignore to build hierarchical reasoning like the human being for mining the contextual structure between words and sentences. In this paper, we propose a Reasoning with Pre-trained Knowledge and Hierarchical Structure (RPKHS) network, which contains a pre-trained knowledge encoder and a hierarchical reasoning encoder. Firstly, our pretrained knowledge encoder aims at reasoning the MWP by using outside knowledge from the pre-trained transformer-based models. Secondly, the hierarchical reasoning encoder is presented for seamlessly integrating the wordlevel and sentence-level reasoning to bridge the entity and context domain on MWP. Extensive experiments show that our RPKHS significantly outperforms state-of-the-art approaches on two large-scale commonly-used datasets, and boosts performance from 77.4% to 83.9% on Math23K, from 75.5 to 82.2% on Math23K with 5-fold cross-validation and from 83.7% to 89.8% on MAWPS. More extensive ablations are shown to demonstrate the effectiveness and interpretability of our proposed method.

MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving

“…Along the path of the MWP solver's development, the pioneer studies use traditional rule-based methods, machine learning methods and statistical methods (Yuhui et al, 2010;Kushman et al, 2014;Shi et al, 2015;Koncel-Kedziorski et al, 2015). Afterwards, inspired by the development of sequence-to-sequence (Seq2Seq) models, MWP solving has been formulated as a neurosymbolic reasoning pipeline of translating language descriptions to mathematical equations with encoder-decoder framework (Wang et al, 2018Zhang et al, 2020b;Yu et al, 2021;Wu et al, 2021a). By fusing hard constraints into decoder (Chiang and Chen, 2018;Liu et al, 2019a;Xie and Sun, 2019;Shen and Jin, 2020;Zhang et al, 2020a), MWP solvers achieve much better performance then.…”

“…Recent works in math word problem (MWP) solving (Wang et al, 2018Liu et al, 2019a;Xie and Sun, 2019;Zhang et al, 2020b;Wu et al, 2020;Qin et al, 2021;Huang et al, 2021;Wu et al, 2021a;Yu et al, 2021;Shen et al, 2021) arrange the pipeline into a sequenceto-sequence framework. In brief, they use deep representation and gradient optimization as well as symbolic constraints to discover discrete symbolic combinations of operators and variants.…”

“…Fundamentally, MWP solving system aims to perform symbolic reasoning by searching through a combinatorial solution space given the text description evidences. Thus, these neurosymbolic methods mainly focus on getting more effective semantic representations Zhang et al, 2020b;Wu et al, 2020Wu et al, , 2021aYu et al, 2021), injecting symbolic constraints (Wang et al, 2018Liu et al, 2019a;Xie and Sun, 2019) and how to align semantic space (text descriptions) and huge combinatorial symbolic space (symbolic solutions) (Qin et al, 2021;Shen et al, 2021;Huang et al, 2021). This line of methods has achieved great success and is still holding the lead in various MWP solving benchmarks (Wang et al, 2017;Zhao et al, 2020;Koncel-Kedziorski et al, 2016).…”

“…to encode quantities. RPKHS(Yu et al, 2021) builds hierarchical reasoning encoder in parallel with PLM encoder. REAL(Huang et al, 2021) proposes a human-like analogical auxiliary learning strategy.…”

An Improved Math Word Problem (MWP) Model Using Unified Pretrained Language Model (UniLM) for Pretraining

“…Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. e train skills in [30][31][32][33] are applied in this paper.…”