Item Parameter Estimation Under Conditions of Test Speededness: Application of a Mixture Rasch Model With Ordinal Constraints

Abstract

When tests are administered underhed time constraints, test performunces can be affected by speededness. Among other consequences, speededness can result in inaccurate parameter estimates in item response theory (IRT) models, especially for items located near the end of tests (Oshima, 1994). This article presents an IRT strategy for reducing contamination in item difficulty estimates due to speededness. Ordinal constraints are applied to a mixture Rasch model (Rost, 1990) so as to distinguish two latent classes of examinees: (a) a "speeded" class, comprised of examinees that had insufficient time to adequately answer end-of-test items, and (b) a "nonspeeded" class, comprised of examinees that had sufficient time to answer all items. The parameter estimates obtained for end-of-test items in the nonspeeded class are shown to more accurately approximate their difficulties when the items are administered at earlier locations on a different form of the test. A mixture model can also be used to estimate the class memberships of individual examinees. In this way, it can be determined whether membership in the speeded class is associated with other student characteristics. Results are reported for gender and ethnicity.

Testing for Aberrant Behavior in Response Time Modeling

“…When tests are given without the objective being to measure the working speed, which is also unrelated to the construct that is supposed to be measured, the purpose of the test is negatively affected by effects of speededness. The performance of the test takers will be affected by effects of test speededness, which can lead to more guessing behavior and inaccurate item parameter estimates (Bolt et al, 2002) For the last type of aberrant behavior, one extreme RT, the percentage of aberrant patterns in the sample highly influenced the results. The detection rates were acceptable when 5% of the RT patterns were simulated to be aberrant, but for higher percentages the detection rates were much lower.…”

“…Schnipke and Scramms (1997) studied rapid guessing, where part of the items show unusually small RTs. Bolt, Cohen, and Wollack (2002) focused on test speededness toward the end of a test. For some respondents who run out of time, one might observe unexpected small RTs during the last part of the test.…”

Parameter Estimation with Mixture Item Response Theory Models: A Monte Carlo Comparison of Maximum Likelihood and Bayesian Methods

“…Use of the MixIRT model in a variety of contexts has been described in detail by a number of authors (Cohen & Bolt, 2005;von Davier & Yamamoto, 2004;von Davier & Rost, 1995;Mislevy & Verhelst, 1990;Rost, 1990;Yamamoto, 1987 classes) which are characterized by different item response models for a particular measure or instrument (Li, et al, 2009). In this context, psychometricians have used MixIRT to detect and characterize differential item functioning (DIF) (Cohen & Bolt, 2005;De Ayala, et al, 2002;Bolt, Cohen & Wollack, 2002, 2001). This simulation study compares the parameter estimation accuracy for two methods of estimation used with MixIRT: Maximum Likelihood Estimation (MLE) and Baysian estimation using the Markov Chain Monte Carlo (MCMC) approach.…”

“…Mixture item response theory (MixIRT) has become an increasingly popular tool for investigating a variety of issues in educational and psychological assessment (Cohen & Bolt, 2005;Bolt, Cohen & Wollack, 2001). Use of the MixIRT model in a variety of contexts has been described in detail by a number of authors (Cohen & Bolt, 2005;von Davier & Yamamoto, 2004;von Davier & Rost, 1995;Mislevy & Verhelst, 1990;Rost, 1990;Yamamoto, 1987 classes) which are characterized by different item response models for a particular measure or instrument (Li, et al, 2009).…”

IRT model fit from different perspectives

“…There has been a recent surge in the use of Bayesian estimation in IRT. Albert (1992), Junker (1999a, 1999b), Bradlow, Wainer, and Wang (1999), Janssen, Tuerlinckx, Meulders, and De Boeck (2000), Béguin and Glas (2001), Glas (2001, 2003), Bolt, Cohen, and Wollack (2002), Sinharay, Johnson, and Williamson (2003), and Wollack, Cohen, and Wells (2003) are only some of the recent examples of application of Bayesian estimation and MCMC algorithms (e.g., Gelman, Carlin, Stern, & Rubin, 2004) to fit complicated psychometric models. There are numerous instances in which classical methods fail, but the Bayesian approach offers a feasible method for assessing model fit.…”