Which is the Effective Way for Gaokao: Information Retrieval or Neural Networks?

Abstract

As one of the most important test of China, Gaokao is designed to be difficult enough to distinguish the excellent high school students. In this work, we detailed the Gaokao History Multiple Choice Questions(GKHMC) and proposed two different approaches to address them using various resources. One approach is based on entity search technique (IR approach), the other is based on text entailment approach where we specifically employ deep neural networks(NN approach). The result of experiment on our collected real Gaokao questions showed that they are good at different categories of questions, i.e. IR approach performs much better at entity questions(EQs) while NN approach shows its advantage on sentence questions(SQs). Our new method achieves state-of-the-art performance and show that it's indispensable to apply hybrid method when participating in the real-world tests.

Improving answer selection with global features

“…For example, Chen, Gao, Shi, Du, and Wen (2014) employ a language model to evaluate whether two sentences have a question‐answer relationship, and Surdeanu, Ciaramita, and Zaragoza (2011) use linguistically motivated features to train a ranking model for selecting correct results. Guo et al (2017) verify that IR approaches perform relatively well on entity questions. In information retrieval models, the statistical global features obtained from QA corpora are widely used, while the implicit semantics between words are not effectively utilized.…”

“…Rather than learning representations of the question and answer candidate separately, researchers have recently introduced various types of attention mechanisms to the answer selection task (Bian, Li, Yang, Chen, & Lin, 2017; Deng et al, 2019; Kim, Kang, & Kwak, 2019; Shen et al, 2018; Tay, Tuan, & Hui, 2018b) that better focus on the relevant parts of the input QA pairs. To enrich the representation of features, researchers have integrated knowledge bases into neural networks and captured more relevant information to improve performance (Deng et al, 2018; Guo et al, 2017; Shijia, Xu, & Xiang, 2018; F. Wang, Wu, Li, & Zhou, 2017; J. Wang, Wang, Zhang, & Yan, 2017; Zhu, Cheng, & Su, 2020). Recently, a new paradigm is to obtain better performance by using huge pre‐trained models (e.g., ELMo, BERT) (Li, Yu, Chen, & Li, 2019; Mozafari, Fatemi, & Nematbakhsh, 2019).…”

“…With the increasing popularity of deep neural networks in natural language processing (Khamparia & Singh, 2019), a large number of answer selection models (Deng et al, 2019; Guo, Zeng, He, Liu, & Zhao, 2017; Liu et al, 2018; Tay, Tuan, & Hui, 2018a; Wang, Hamza, & Florian, 2017) have achieved impressive results. However, most of the current models focus on local features generated by QA pairs, while global information contained in the question and its answer candidates (the QA corpus) lacks research.…”

GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation

“…There exist some datasets, collected from standard exams/tests, including English datasets RACE (Lai et al, 2017), DREAM (Sun et al, 2019), ARC (Clark et al, 2018), ReClor , etc., and Chinese datasets C3 (Sun et al, 2020), MCQA (Guo et al, 2017), GeoSQA (Huang et al, 2019) and MedQA etc. Some of these datasets are of specific domains.…”

A Hybrid Framework for Problem Solving of Comparative Questions

“…Other similar works include geographical and history Gaokao [33]- [35], since some of our experimental datasets come from geographical examinations. As an attempt to answer multiple-choice questions in history Gaokao, Cheng et al [33] proposed a three-stage framework including retrieving, ranking, and filtering concept and quote pages, as well as Guo et al [34] proposed a permanent-provisional memory network. The most relevant work is by Ding et al [35] on multiple-choice questions in geographical Gaokao.…”