A New Statistic for Detection of Aberrant Answer Changes

Abstract

As noted by Fremer and Olson, analysis of answer changes is often used to investigate testing irregularities because the analysis is readily performed and has proven its value in practice. Researchers such as Belov, Sinharay and Johnson, van der Linden and Jeon, van der Linden and Lewis, and Wollack, Cohen, and Eckerly have suggested several statistics for detection of aberrant answer changes. This article suggests a new statistic that is based on the likelihood ratio test. An advantage of the new statistic is that it follows the standard normal distribution under the null hypothesis of no aberrant answer changes. It is demonstrated in a detailed simulation study that the Type I error rate of the new statistic is very close to the nominal level and the power of the new statistic is satisfactory in comparison to those of several existing statistics for detecting aberrant answer changes. The new statistic and several existing statistics were shown to provide useful information for a real data set. Given the increasing interest in analysis of answer changes, the new statistic promises to be useful to measurement practitioners.

The use of item scores and response times to detect examinees who may have benefited from item preknowledge

“…The use of was found, in our simulations, to lead to low power and inadvertent flagging of those who performed worse or slower on the compromised items, and is not considered further. Also, Sinharay, Duong, and Wood (2017) emphasized the need to use one‐sided hypothesis testing in detection of test fraud.…”

Detecting Fraudulent Erasures at an Aggregate Level

“…Erasure analysis was also performed at the individual level using the L-index (Sinharay, Duong, & Wood, 2017). The values of the L-index agree with the values of EDI g , EDI g N , and EDI g A for the data set.…”

“…There are several limitations of this article and, consequently, several related topics can be further investigated. First, it is possible to extend other indices for detection of fraudulent erasures for individual examinees including those suggested by Sinharay and Johnson (2017), Sinharay et al (2017), and van der Linden and Lewis (2015) to the group level and a future study may compare the extensions suggested in this article to extensions of other individual-level statistics for detecting fraudulent erasures. Second, while our simulation study was detailed, it is possible to perform more simulations, possibly with other IRT models.…”

“…To address the increasing interest in practice on erasure analysis, there has been an upswing in research on the topic. Recently, researchers such as Belov (2015), Sinharay et al (2017), Sinharay and Johnson (2017), van der Linden and Jeon (2012), van der Linden and Lewis (2015), Wollack et al (2015), and Wollack and Eckerly (2017) presented new statistics for individual-level or group-level erasure analysis. Sinharay et al (2017) performed a comprehensive comparison of several of these statistics at the individual level—they found the EDI (Wollack et al, 2015) and their suggested statistic L-index, which is based on the likelihood ratio statistic, to have performed the best.…”

“…However, erasures essentially mean answer changes (ACs), and computer-based tests (CBTs) may also suffer from fraudulent ACs. Tiemann and Kingston (2014) and Sinharay, Duong, and Wood (2017) provided examples of CBTs in which ACs are allowed-fraudulent ACs can definitely occur for such tests. Wollack, Cohen, and Eckerly (2015) suggested the erasure detection index (EDI) to detect fraudulent erasures for individual examinees.…”