On the Usefulness of a Multilevel Logistic Regression Approach to Person-Fit Analysis

Abstract

The logistic person response function (PRF) models the probability of a correct response as a function of the item locations. Reise (2000) proposed to use the slope parameter of the logistic PRF as a person-fit measure. He reformulated the logistic PRF model as a multilevel logistic regression model and estimated the PRF parameters from this multilevel framework. An advantage of the multilevel framework is that it allows relating person fit to explanatory variables for person misfit/fit. We critically discuss Reise's approach. First, we argue that often the interpretation of the PRF slope as an indicator of person misfit is incorrect. Second, we show that the multilevel logistic regression model and the logistic PRF model are incompatible, resulting in a multilevel person-fit framework, which grossly violates the bivariate normality assumption for residuals in the multilevel model. Third, we use a Monte Carlo study to show that in the multilevel logistic regression framework estimates of distribution parameters of PRF intercepts and slopes are biased. Finally, we discuss the implications of these results and suggest an alternative multilevel regression approach to explanatory person-fit analysis. We illustrate the alternative approach using empirical data on repeated anxiety measurements of cardiac arrhythmia patients who had a cardioverter-defibrillator implanted.

Some thoughts on analytical choices in the scaling model for test scores in international large-scale assessment studies

“…In a further alternative hybrid approach of design-based inference and model-based inference (see Ståhl et al, 2016), subjects can additionally be weighted by including weights ν P,n according to their fit to a statistical model. For example, model-based student-specific weights ν P,n can be derived according to their fit to the scaling model (person fit; see Conijn et al, 2011;Hong & Cheng, 2019;Raiche et al, 2012;Schuster & Yuan, 2011). In such an approach, students whose item responses are atypical with respect to the IRT model (e.g., non-scalable students; see Haertel, 1989) would be downweighted compared to students whose item responses are consistent with the IRT model.…”

“…Moreover, the ability variable θ could also be redefined in a scaling model in which item responses and response times load on θ, resulting in a purified latent variable for speed (Costa et al, 2021). Furthermore, measurement models could also involve an additional student latent variable α n that characterizes person fit (Conijn et al, 2011;Ferrando, 2019;Raiche et al, 2012):…”

Reflections on Analytical Choices in the Scaling Model for Test Scores in International Large-Scale Assessment Studies

“…Moreover, the ability variable θ could also be redefined in a scaling model in which item responses and response times load on θ, resulting in a purified latent variable for speed (Costa et al, 2021). Furthermore, measurement models could also involve an additional student latent variable α that characterizes person fit (Conijn et al, 2011;Ferrando, 2019;Raiche et al, 2012)…”

Assessing Person Fit With the Information Matrix Test

“…Several tests of person fit are aimed at the detection of trait changes between subsets of items; see Klauer and Rettig (1990) and Glas and Dagohoy (2007) for example. The multilevel logistic regression approach of Reise (2000) also evaluates the constancy of θ over the test, but see Conijn, Emons, van Assen, and Sijtsma (2011) for a critical review of this approach.…”