Can Examinees Use a Review Option to Obtain Positively Biased Ability Estimates on a Computerized Adaptive Test?

Abstract

Part of the controversy about allowing examinees to review and change answers to previous items on computerized adaptive tests (CATs) centers on a strategy for obtaining positively biased ability estimates attributed to Wainer (1993) in which examinees intentionally answer items incorrectly before review and to the best of their abilities upon review. Our results, based on both simulated and live testing data, showed that there were instances in which the Wainer strategy yielded inflated ability estimates as well as instances in which it yielded deflated ability estimates. The success of the strategy in inflating ability estimates depended on the ability estimation method used (maximum likelihood versus Bayesian), the examinee's true ability level, the standard error of the ability estimate, the examinee's ability to implement the strategy, and the type of decision made from the ability estimate. We discuss approaches to dealing with the Wainer strategy in operational CAT settings.

Limiting Answer Review and Change on Computerized Adaptive Vocabulary Tests: Psychometric and Attitudinal Results

“…Arguments against review on CATs have focused primarily on added complications in administration and scoring algorithms, decreased validity and precision resulting from the use of clever answer response strategies, and substantial increases in testing time. Although these problems are serious potential obstacles to overcome, evidence from this study and others (e.g., Stocking, 1997, Vispoel et al, 1999 indicates that these problems may not be insurmountable if review opportunities are limited and special efforts are made to streamline CATs for maximum efficiency.…”

“…The tests were adaptive within and between blocks, and all tests were terminated at 40 items. The tests were scored using EAP Bayesian methods as previous research had shown that EAP methods were much less susceptible to score distortion resulting from use of the Wainer strategy than were ML methods (Vispoel et al, 1999). After each item was administered, the computer recorded the item sequence number, the item pool number, the item parameters (a, b, and c), the examinee's answer, the correct answer, the examinee's item score (1,0), the examinee's current (EAP) ability estimate, the posterior variance for that ability estimate, and the time spent answering the item.…”

“…If the items are sufficiently mismatched to the examinee's true ability level, he or she may obtain a positively biased ability estimate or a spuriously high ability estimate resulting from high measurement error. Vispoel, Rocklin, Wang, and Bleiler (1999) evaluated these possibilities using both simulated data and data obtained from live examinees coached in implementing the Wainer strategy on an operational vocabulary CAT based on the threeparameter Logistic model. The results revealed that the Wainer strategy had a much more detrimental effect on maximum likelihood (ML)-based ability estimates and standard errors than on expected a posteriori (EAP) ability estimates and their errors.…”

Evaluating the Comparability of Paper‐ and Computer‐Based Science Tests Across Sex and SES Subgroups

“…Nonetheless, the use of computer‐based assessments presents some challenges. Previous research indicates that graphical or complex displays (Mazzeo & Harvey, 1988), speededness (Mead & Drasgow, 1993), screen/font size and resolution (Bridgeman, Lennon, & Jackenthal, 2003; Oetjen & Ziefle, 2009), the (in)ability to revise/review previous responses (Revuelta, Ximenez, & Olea, 2003; Vispoel, Rocklin, Wang, & Bleiler, 1999), and familiarity or practice time with computers (Taylor, Kirsch, Jamieson, & Eignor, 1999; Wang, Jiao, Young, Brooks, & Olson, 2008) may all affect student performance on computer‐based assessments. Moreover, full CBT implementation can be difficult due to limited resources at either the national or school level.…”

Can Examinees Use Judgments of Item Difficulty to Improve Proficiency Estimates on Computerized Adaptive Vocabulary Tests?

“…Two gaming strategies, typically labeled the Wainer and Kingsbury strategies, have been examined in the research literature. When using the Wainer strategy (see Gershon & Bergstrom, 1995;Stocking, 1997;Vispoel, Rocklin, Wang, & Bleiler, 1999;Wainer, 1993), an examinee would intentionally answer all items incorrectly on the first pass and then answer each item to the best of his or her ability upon review. If the items are sufficiently mismatched to the examinee's true proficiency level, he or she may obtain a positively biased proficiency estimate or a spuriously high proficiency estimate resulting from high measurement error.…”