Performance of Person‐Fit Statistics Under Model Misspecification

Abstract

In educational and psychological measurement, a person‐fit statistic (PFS) is designed to identify aberrant response patterns. For parametric PFSs, valid inference depends on several assumptions, one of which is that the item response theory (IRT) model is correctly specified. Previous studies have used empirical data sets to explore the effects of model misspecification on PFSs. We further this line of research by using a simulation study, which allows us to explore issues that may be of interest to practitioners. Results show that, depending on the generating and analysis item models, Type I error rates at fixed values of the latent variable may be greatly inflated, even when the aggregate rates are relatively accurate. Results also show that misspecification is most likely to affect PFSs for examinees with extreme latent variable scores. Two empirical data analyses are used to illustrate the importance of model specification.

Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals

“…It is also more powerful for the longer test lengths, but caution in interpretation is warranted because of a few inflated Type I error rates under the null for each statistic. As for M 2 , it has low-to-moderate power, which is consistent with prior research (Hong et al, in press). Similarly, prior research has found S − X 2 to have low-to-moderate power in detecting this form of misspecification, which is consistent with the results in Table 4.…”

Using item scores and response times in person‐fit assessment

“…Other combinations of statistics could also be explored. Third, it would be useful to examine the impact of model misspecification on the performance of the new statistics (Hong et al, 2020; Meijer & Tendeiro, 2012). Fourth, the statistics could be extended to multidimensional tests (Albers et al, 2016; Drasgow et al, 1991; Hong et al, 2021) and/or tests with polytomous items (Hong et al, 2021; Sinharay, 2016a).…”