R2de

Abstract

The main objective of exams consists in performing an assessment of students' expertise on a specific subject. Such expertise, also referred to as skill or knowledge level, can then be leveraged in different ways (e.g., to assign a grade to the students, to understand whether a student might need some support, etc.). Similarly, the questions appearing in the exams have to be assessed in some way before being used to evaluate students. Standard approaches to questions' assessment are either subjective (e.g., assessment by human experts) or introduce a long delay in the process of question generation (e.g., pretesting with real students). In this work we introduce R2DE (which is a Regressor for Difficulty and Discrimination Estimation), a model capable of assessing newly generated multiple-choice questions by looking at the text of the question and the text of the possible choices. In particular, it can estimate the difficulty and the discrimination of each question, as they are defined in Item Response Theory. We also present the results of extensive experiments we carried out on a real world large scale dataset coming from an e-learning platform, showing that our model can be used to perform an initial assessment of newly created questions and ease some of the problems that arise in question generation.

Historical Perspectives on Score Comparability Issues Raised by Innovations in Testing

“…As with examinee covariates, interest in the relationship between item characteristics and various ancillary data extends beyond the domain of score comparability. Applications include difficulty or item parameter prediction (e.g., Baldwin et al., 2004; Collis et al., 1995; Hall & Ansley, 2008; Irvine et al., 1990; Mislevy, 1988; Nungester & Vass, 1985; Scheuneman et al., 1991; Stowe, 2002; Swaminathan et al., 2003; Wang & Jiao, 2011; Xie, 2019); response time prediction (e.g., Halkitis et al., 1996; Parshall et al., 1994; Smith, 2000; Swanson et al., 2001; Baldwin et al., 2021); evaluation of automatically generated items (Leo et al., 2019; Kurdi, 2020; Benedetto et al., 2020); item pretest survival prediction (Ha et al., 2019, Yaneva et al., 2020); response process complexity estimation (Yaneva et al., 2021); and differential item functioning detection (Sinharay et al., 2009). Nevertheless, while researchers have sought to capitalize on item covariates to improve a broad range of activities, they have not been widely used to facilitate score comparability.…”