Semantic similarity of distractors in multiple-choice tests

Abstract

Mitkov and Ha (2003) and Mitkov et al. (2006) offered an alternative to the lengthy and demanding activity of developing multiple-choice test items by proposing an NLP-based methodology for construction of test items from instructive texts such as textbook chapters and encyclopaedia entries. One of the interesting research questions which emerged during these projects was how better quality distractors could automatically be chosen. This paper reports the results of a study seeking to establish which similarity measures generate better quality distractors of multiple-choice tests. Similarity measures employed in the procedure of selection of distractors are collocation patterns, four different methods of WordNetbased semantic similarity (extended gloss overlap measure, Leacock and Chodorow's, Jiang and Conrath's as well as Lin's measures), distributional similarity, phonetic similarity as well as a mixed strategy combining the aforementioned measures. The evaluation results show that the methods based on Lin's measure and on the mixed strategy outperform the rest, albeit not in a statistically significant fashion.

Multiple-Choice Item Generator for Narrative and Declarative Texts

“…To determine the word similarity score, Wordnet Similarity is used to perform Lin"s method of measuring semantic relatedness [7]. In determining similarity score between two phrases, a maximal bipartite graph is constructed first.…”

Distractor Quality Evaluation in Multiple Choice Questions

“…Table 2 shows that the ranking model gives higher balance between recall and precision than individual measures, regardless of the corpus. In particular it gives a higher precision than other measures and better results than WordNet-based measures, used by [11].…”

“…WordNet also contains named entities for a few kinds of entities (large cities, countries, continents...). We use the four measures selected by [11]: the extended gloss overlap measure based on textual similarity between the glosses of two concepts; Leacock and Chodorow's measure based on the shortest path between concepts; and Jiang and Conrath's and Lin's measures, both based on information content [14]. Terms can have multiple senses, so they can be associated with multiple synsets.…”

“…We focused our work on short options, expressed by chunks (lowest-level phrases) and named entities. Previous research on automatic distractor selection had more restrictions on the type of options [6,11], or was dedicated to a specific domain [6]. To our knowledge, it is the first attempt to evaluate automatically the semantic quality of MCQ distractors.…”

Information-based methods for evaluating the semantics of automatically generated test items

“…Information on semantic relatedness will enhance the quality and usability of generated item pools thereby leading to the selection of higher quality distractors [8] using semantic descriptions of the item content that, in turn, could help predict the item difficulty level, [9] the quantification of language diversity in the generated item pool, the use of similarity-based theory to control the difficulty of the multiple-choice questions -MCQs [10] and a better understanding what makes an item difficult for a group of students. However, visual evaluation of similarity and semantic relatedness is subjective and therefore ineffective and could be impractical for large set of automatically generated items, Unfortunately, quantifying item similarity is challenging because prior semantic knowledge about the assessment tasks is often not available.…”