Detection of Test Speededness Using Change-Point Analysis

Abstract

Change-point analysis (CPA) is a well-established statistical method to detect abrupt changes, if any, in a sequence of data. In this paper, we propose a procedure based on CPA to detect test speededness. This procedure is not only able to classify examinees into speeded and non-speeded groups, but also identify the point at which an examinee starts to speed. Identification of the change point can be very useful. First, it informs decision makers of the appropriate length of a test. Second, by removing the speeded responses, instead of the entire response sequence of an examinee suspected of speededness, ability estimation can be improved. Simulation studies show that this procedure is efficient in detecting both speeded examinees and the speeding point. Ability estimation is dramatically improved by removing speeded responses identified by our procedure. The procedure is then applied to a real dataset for illustration purpose.

Joint Modeling of Ability and Differential Speed Using Responses and Response Times

“…The general idea is that the time limit influences the probability of an item being answered, without considering RT information. Shao and Cheng (2015) considered a change-point model to identify speeded test takers and considered removing the speeded responses to improve ability estimation. Given the RT and response information, the joint model for ability and speed, using a latent growth model for speed, can provide insight into test speededness.…”

Modeling Item Revisit Behavior: The Hierarchical Speed–Accuracy–Revisits Model

“…One of the most frequently utilized type of information from process data is the time that examinees take to answer a question. Response times (RTs) are employed in a wide variety of applications in psychometrics such as detecting aberrant response behavior (Shao, 2016; van der Linden et al, 1999; van der Linden & Guo, 2008), test assembly (van der Linden, 2011), modeling of missing responses (Pohl et al, 2019; Ulitzsch et al, 2019), item selection in computerized adaptive testing (e.g., Fan et al, 2012). In addition to the operational use of RT, a range of modeling approaches have been developed that incorporate RT into the measurement models as ancillary information with the goal to increase the precision of ability estimates.…”

Robust maximum marginal likelihood (RMML) estimation for item response theory models

“…Some researchers control for aberrant responses by directly modeling these types of complex processes (Shao, Li, & Cheng, 2016;Wang, Xu, & Shang, 2018;Yamamoto & Everson, 2003). In the psychological literature, others consider careless responses a Bspecial case^in a more general modeling framework of response styles (Böckenholt, 2017;Falk & Cai, 2016;Wetzel & Carstensen, 2017).…”