Robust Maximum Marginal Likelihood (RMML) Estimation for Item Response Theory Models

Abstract

Self-report data are common in psychological and survey research. Unfortunately, many of these samples are plagued with careless responses due to unmotivated participants. The purpose of this study is to propose and evaluate a robust estimation method in order to detect careless, or unmotivated, responders while leveraging Item Response Theory (IRT) person fit statistics. First, we outline a general framework for robust estimation specific for IRT models.Subsequently, we conduct a simulation study covering multiple conditions to evaluate the performance of the proposed method. Ultimately, we show how robust maximum marginal likelihood (RMML) estimation significantly improves detection rates for careless responders and reduce bias in item parameters across conditions. Furthermore, we apply our method to a real dataset to illustrate the utility of the proposed method. Our findings suggest that robust estimation coupled with person fit statistics offers a powerful procedure to identify careless respondents for further review, and to provide more accurate item parameter estimates in presence of careless responses.

Robust Estimation for Response Time Modeling

“…However, there is a trade‐off between bias and efficiency. This study extends previous work in the robust estimation framework designed for item responses to response times (Hong & Cheng, 2019; Schuster & Yuan, 2011). Moreover, our paper suggests that including BW functions for robust estimators would be appropriate for response time modeling with a tuning parameter set to .…”

“…In the context of IRT, robust estimation has been employed to address similar concerns (Hong & Cheng, 2019; Schuster & Yuan, 2011; Wainer & Thissen, 2008). M‐estimation has been shown to improve latent trait or ability estimates when response vectors are contaminated with careless responses or test speededness (Schuster & Yuan, 2011; Wainer & Thissen, 2008; Mislevy & Bock, 1982).…”

“…M‐estimation has been shown to improve latent trait or ability estimates when response vectors are contaminated with careless responses or test speededness (Schuster & Yuan, 2011; Wainer & Thissen, 2008; Mislevy & Bock, 1982). More recently, M‐estimation has also been found to improve the ability to recover structural or item parameters using the same general framework (Hong & Cheng, 2019). Employing robust estimation has also been shown to improve the ability to detect deviating behavior compared to no downweighting or data removal (Hong & Cheng, 2019; Kim, Reise, & Bentler, 2018; Sinharay, 2018).…”

“…More recently, M‐estimation has also been found to improve the ability to recover structural or item parameters using the same general framework (Hong & Cheng, 2019). Employing robust estimation has also been shown to improve the ability to detect deviating behavior compared to no downweighting or data removal (Hong & Cheng, 2019; Kim, Reise, & Bentler, 2018; Sinharay, 2018).…”

An overview of applied robust methods

“…In practice, item parameters may not be known in advance, and guessing or careless responses will also interfere with their estimation. Hong and Cheng (2019) developed M‐estimation for item parameters of the graded response model (GRM) with polytomous items. The individual terms in equation (15) then become the derivatives of the marginal likelihood function of the GRM with respect to item parameters.…”