Information-based methods for evaluating the semantics of automatically generated test items

Abstract

Multiple-choice questions are the popular type of test items that are used for testing the knowledge of health-science students in north America and elsewhere. The motivation of this article is to present the recent advances in the automatic item generation (AIG) and to propose a novel unsupervised approach that extends the information-based Compositional Distributional Semantic Model (CDSM) to measure the semantic relatedness among the pool of automatically generated items. We have used operational item bank from the medical science domain for developing the CDSM and demonstrated our approach using the concepts from AIG research. We illustrated our approach using eleven item models from the medical education domain, and discussed the possible applications to advance the AIG research.

Development of the automatic item generation system for the diagnosis of misconceptions about force and laws of motion

“…Item models with a low CSI average and a high SD will generate items that are highly different from one another, also known as variants (Gierl & Lai, 2013). We used the study by Latifi et al (2017), which found that the item model that generated isomorphs will have a CSI average of over 0.70 and a SD of lower 0.10. While item model that generated variants will have a CSI average of lower 0.70 and an SD of over 0.10.…”

The Effect of Option Homogeneity in Multiple-Choice Items

“…CSI is a text-vector indexing technique which measures the similarity between two vectors of co-occurring text. A distance score is used to compare the similarity of the text (Becker & Kao, 2009; Latifi, Gierl, Wang, Lai, & Wang, 2016). The wording of the key and each option were compared with a root word dictionary.…”