Generating Distractors for Reading Comprehension Questions from Real Examinations

Abstract

We investigate the task of distractor generation for multiple choice reading comprehension questions from examinations. In contrast to all previous works, we do not aim at preparing words or short phrases distractors, instead, we endeavor to generate longer and semantic-rich distractors which are closer to distractors in real reading comprehension from examinations. Taking a reading comprehension article, a pair of question and its correct option as input, our goal is to generate several distractors which are somehow related to the answer, consistent with the semantic context of the question and have some trace in the article. We propose a hierarchical encoderdecoder framework with static and dynamic attention mechanisms to tackle this task. Specifically, the dynamic attention can combine sentence-level and word-level attention varying at each recurrent time step to generate a more readable sequence. The static attention is to modulate the dynamic attention not to focus on question irrelevant sentences or sentences which contribute to the correct option. Our proposed framework outperforms several strong baselines on the first prepared distractor generation dataset of real reading comprehension questions. For human evaluation, compared with those distractors generated by baselines, our generated distractors are more functional to confuse the annotators.

Automatic Distractor Generation for Multiple Choice Questions in Standard Tests

“…They employed the hierarchical encoder-decoder network as the base model and used the dynamic attention mechanism to generate the long distractors. Zhou et al (2019) further strengthened the interaction between the question and the passage based on the model of (Gao et al, 2018).…”

“…These word-level or entity-level methods can only generate short distractors and do not apply to semantic-rich and long distractors for RACE-like MCQs. Recently, generating longer distractors has been explored in a few studies (Zhou et al, 2019;Gao et al, 2018). For example, Gao et al (2018) proposes a sequence-to-sequence based model, which leverages the attention mechanism to automatically generate distractors from the reading passages.…”

“…Recently, generating longer distractors has been explored in a few studies (Zhou et al, 2019;Gao et al, 2018). For example, Gao et al (2018) proposes a sequence-to-sequence based model, which leverages the attention mechanism to automatically generate distractors from the reading passages. However, these methods mainly focus on the relation between the distractor and the passage and fail to comprehensively model the interactions among the passage, question and correct answer which helps to ensure the incorrectness of the generated distractors.…”

“…Recently, neural network based and data-driven solutions emerge. Gao et al (2018) proposed an end-to-end solution focusing on distractor generation for MCQs in standard English tests. They employed the hierarchical encoder-decoder network as the base model and used the dynamic attention mechanism to generate the long distractors.…”

“…Rather than being a trivial wrong answer, the distractor should have the plausibility which confuses learners who did not master the knowledge (Liang et al, 2018;Qiu et al, 2019). A good distractor should be grammatically correct given the question and semantically consistent with the passage context of the question (Gao et al, 2018). Meanwhile, the question composers need to enhance the plausibility of the distractor without hurting its inherent incorrectness.…”