Comparing the Difficulty of Examination Subjects with Item Response Theory

Abstract

Methods are presented for comparing grades obtained in a situation where students can choose between different subjects. It must be expected that the comparison between the grades is complicated by the interaction between the students' pattern and level of proficiency on one hand, and the choice of the subjects on the other hand. Three methods based on item response theory (IRT) for the estimation of proficiency measures that are comparable over students and subjects are discussed: a method based on a model with a unidimensional representation of proficiency, a method based on a model with a multidimensional representation of proficiency, and a method based on a multidimensional representation of proficiency where the stochastic nature of the choice of examination subjects is explicitly modeled. The methods are compared using the data from the Central Examinations in Secondary Education in the Netherlands. The results show that the unidimensional IRT model produces unrealistic results, which do not appear when using the two multidimensional IRT models. Further, it is shown that both the multidimensional models produce acceptable model fit. However, the model that explicitly takes the choice process into account produces the best model fit.

Estimation of the Proportion of Underachieving Students in Compulsory Secondary Education in Spain: An Application of the Rasch Model

“…Taking into account that school grades do not constitute a validated test, a deeper analysis of the fit of the courses has been conducted, based on the inter-subject comparability approach (Tasmanian Qualification Authority, 2006 , 2007 ; Coe, 2008 ; Korobko et al, 2008 ). Table 1 shows the courses analyzed, the indices of fit, and the item-scale correlation.…”

Relationship between parent involvement and academic achievement through metacognitive strategies: A multiple multilevel mediation analysis

“…Scores of subjects of each grade presented a high reliability, with Cronbach's alpha values of .93 for first‐grade participants, and .94 for the second‐grade participants. In the present study, all subjects were compulsory for students; thus, it was not possible for choice of examination to affect the measurement of the latent construct (Korobko, Glas, Bosker, & Luyten, ).…”

Latent Variable Modelling with Non-Ignorable Item Non-Response: Multigroup Response Propensity Models for Cross-National Analysis

“…In recent years such models have been studied most actively in the context of item response theory modelling of data from educational and psychological testing, where the items are categorical (often binary) responses to test questions, the substantive latent variables η describe respondents' abilities or psychological characteristics and non-ignorable non-response may arise when respondents omit questions by choice or because of running out of time, for reasons which may be related to η (Mislevy and Wu, 1996;Mislevy, 2016). Developments, applications and evaluations of latent response propensity models in this field include Glas and Pimentel (2008), Korobko et al (2008), Bertoli-Barsotti and Punzo (2013), Pohl et al (2014) and Köhler et al (2015a,b). Of particular relevance for the cross-national focus of our paper is Rose et al (2010), who described different approaches to modelling cross-national data from educational tests with non-response, and carried out a multigroup analysis with latent response propensities for data from the Programme for International Student Assesssment in 30 countries.…”