Robust Estimation of Ability and Mental Speed Employing the Hierarchical Model for Responses and Response Times

Abstract

Van der Linden's hierarchical model for responses and response times can be used in order to infer the ability and mental speed of test takers from their responses and response times in an educational test. A standard approach for this is maximum likelihood estimation. In real‐world applications, the data of some test takers might be partly irregular, resulting from rapid guessing or item preknowledge. The maximum likelihood estimator is not robust against contamination with irregular data. In this article, we propose a robust estimator of ability and mental speed. The estimator consists of two steps. In the first step, the mental speed is estimated with the estimator of Gervini and Yohai that ignores outlying response times. In the second step, the ability is estimated with an M‐estimator that down weights unusual responses given at unusual response times. This is achieved by combining the hard‐rejection weights of Gervini and Yohai with the M‐estimator suggested by Croux and Haesbroeck for the logistic regression model. The proposed estimator is consistent, almost as efficient as the maximum likelihood estimator in uncontaminated data and robust in contaminated data. The performance of the estimator is analyzed in a simulation study and an empirical example.

Generating Models for Item Preknowledge

“…There are many different ways in which this can be done. For example, Meijer and Sotaridona (2006) and Ranger, Kuhn, and Wolgast (2021) simulated the compromised RT to be proportional to the secure RT. Under this model, $$\begin{equation} g(t_{ji}) = {\begin{cases} 1 &\text{if }\; t_{ji} = \pi \times t_{ji}^*, \\ 0 &\text{otherwise}, \end{cases}} \end{equation}$$where $$t_{ji}^*$$ is the hypothetical RT of examinee $$j$$ on item $$i$$ had it been secure, and $$\pi \in [0, 1]$$ is the proportion parameter for which smaller values indicate a greater preknowledge effect.…”

“…Meijer and Sotaridona (2006) set $$\pi =.25$$ and .5, while Ranger et al. (2021) used a more conservative value of $$\pi =.6$$. In the real data, the average proportions comparing items before and after compromise were .759, .737, .548, and .518, respectively.…”

“…There are many different ways in which this can be done. For example, Meijer and Sotaridona (2006) and Ranger, Kuhn, and Wolgast (2021) simulated the compromised RT to be proportional to the secure RT. Under this model,…”

“…where t * ji is the hypothetical RT of examinee j on item i had it been secure, and π ∈ [0, 1] is the proportion parameter for which smaller values indicate a greater preknowledge effect. Meijer and Sotaridona (2006) set π = .25 and .5, while Ranger et al (2021) used a more conservative value of π = .6. In the real data, the average proportions comparing items before and after compromise were .759, .737, .548, and .518, respectively.…”