Methodology Review: Evaluating Person Fit

Abstract

Person-fit methods based on classical test theory-and item response theory (IRT), and methods investigating particular types of response behavior on tests, are examined. Similarities and differences among person-fit methods and their advantages and disadvantages are discussed. Sound person-fit methods have been derived for the Rasch model. For other IRT models, the empirical and theoretical distributions differ for most person-fit statistics when used with short and moderate length tests. The detection rate of person-fit statistics depends on the type of misfitting item-score patterns, test length, and trait levels. The usefulness of person-fit statistics for improving measurement depends on the application.

Person fit assessment using the PerFit package in R

“…A misfit is found when the hypothesis is not supported by the observed data. Examples of IRT-based person-fit statistics include Wright and Stone's (1979) Meijer and Sijtsma (2001) for an extensive review and discussion on several PFSs.…”

A Statistical Procedure for Testing Unusually Frequent Exactly Matching Responses and Nearly Matching Responses

“…Traditional investigation into this issue has emphasized answer copying between specific pairs of examinees under investigation (e.g., Angoff, 1974;Holland, 1996;Lewis & Thayer, 1998;van der Linden & Sotaridona, 2006;Wollack, 1997). Residual analysis and person-fit statistics have been proposed to examine aberrant response patterns, or more specially, item preknowledge (e.g., Drasgow, Levine, & Williams, 1985;Karabatsos, 2003;McLeod & Lewis, 1999;McLeod, Lewis, & Thissen, 2003;Meijer & Sijtsma, 2001;Segall, 2002;Shu, Henson, & Luecht, 2013). Residual analysis of both response and timing data has also been considered for the same purpose (e.g., van der Linden & Guo, 2008).…”

IRT model fit from different perspectives

“…Various researchers have found that even when true ability is used to calculate z l , the conditional null distributions of z l are neither standard normal nor consistent across different θ values. Instead, the distributions are negatively skewed, and often leptokurtic (Meijer & Sijtsma, 2001;Molenaar & Hoijtink, 1990;Nering, 1995). When the true θ is replaced by its estimateθ in the calculation of z l , the problems were exacerbated; variable errors at different levels of the estimated ability cause the variance of z l to be very inconsistent across the θ levels.…”

“…These item score patterns should be detected because scores of such persons may not be adequate descriptions of their trait level. This area of research is commonly referred to as person fit research, and the majority of the research on person fit has concentrated on the development of statistics that can be used to identify nonfitting response vectors (van Krimpen-Stoop & Meijer, 1999;Meijer & Sijtsma, 2001). A response pattern is considered nonfitting or aberrant if it is found to be unlikely given the model according to a person fit statistics (PFS).…”

“…Finally, some conclusions are drawn, and some suggestions for further research are given. Meijer and Sijtsma (2001) have presented a comprehensive overview of person fit statistics that are widely used in practice. A number of these statistics will be used below.…”