Learning to Distract: A Hierarchical Multi-Decoder Network for Automated Generation of Long Distractors for Multiple-Choice Questions for Reading Comprehension

Abstract

The task of generating incorrect options for multiple-choice questions is termed as distractor generation problem. The task requires high cognitive skills and is extremely challenging to automate. Existing neural approaches for the task leverage encoder-decoder architecture to generate long distractors. However, in this process two critical points are ignored-firstly, many methods use Jaccard similarity over a pool of candidate distractors to sample the distractors. This often makes the generated distractors too obvious or not relevant to the question context. Secondly, some approaches did not consider the answer in the model, which caused the generated distractors to be either answer-revealing or semantically equivalent to the answer. In this paper, we propose a novel Hierarchical Multi-Decoder Network (HMD-Net) consisting of one encoder and three decoders, where each decoder generates a single distractor. To overcome the first problem mentioned above, we include multiple decoders with a dis-similarity loss in the loss function. To address the second problem, we exploit richer interaction between the article, question, and answer with a SoftSel operation and a Gated Mechanism. This enables the generation of distractors that are in context with questions but semantically not equivalent to the answers. The proposed model outperformed all the previous approaches significantly in both automatic and manual evaluations. In addition, we also consider linguistic features and BERT contextual embedding with our base model which further push the model performance. CCS CONCEPTS • Computing methodologies → Natural language generation.

Musnad Inscriptions from Jabal al-Samma at Najran Region, South West of Saudi Arabia

“…In reading comprehension, participants may evaluate the generated candidates in comparative or quantitative methods Maurya and Desarkar, 2020). The former typically involve the selection of distractors based on specific objectives such as confusion criteria by showing list of generated candidates with ground truth answer -without telling the answer to participants and let participant select the most suitable option.…”

“…Creating multiple-choice questions manually is one of the most labor-intensive task for educators (Ch and Saha, 2018), since questions need to include plausible false options, known as distractors, able to confuse the examinee. To generate distractors, various approaches are utilized, including learning-based (Liang et al, 2018) that rank options according to a set of features, advanced deep neural networks (Maurya and Desarkar, 2020), and pretrained large language models (Chiang et al, 2022).…”