Revising Item Responses in Computerized Adaptive Tests: A Comparison of Three Models

Abstract

Interest in the application of large-scale computerized adaptive testing has focused attention on issues that arise when theoretical advances are made operational. One such issue is that of the order in which exaniinees address questions within a test or separately timed test section. In linear testing, this order is entirely under the control of the examinee, who can look ahead at questions and return and revise answers to questions. Using simulation, this study investigated three models that permit restricted examinee control over revising previous answers in the context of adaptive testing. Even under a worstcase model of examinee revision behavior, two of the models of permitting item revisions worked well in preserving test fairness and accuracy. One model studied may also preserve some cognitive processing styles developed by examinees for a linear testing environment.

Can Examinees Use a Review Option to Obtain Positively Biased Ability Estimates on a Computerized Adaptive Test?

“…A third approach to dealing with the Wainer strategy is to restrict review opportunities. Stocking (1997), for example, obtained acceptable results for bias and precision of ability estimates when review within CATs was limited to items within separately timed blocks or to items linked to a common stimulus (e.g., reading passage, table, graph). Such techniques may provide a reasonable compromise in satisfying examinees' desires for review and in meeting the test users' goal of obtaining valid ability estimates in an efficient manner.…”

“…The authors concluded that cheating on CATs is a risky business and something that can be detected and prevented during test administration by altering test targeting. Stocking (1997) used data simulation and computerized adaptive versions of the Graduate Records Examination to evaluate the effectiveness of the Wainer strategy in comparison to three alternative models in which item review was allowed or partially allowed. The three models permitted: (a) revisions to a limited number of items, (b) revisions to all items within separately timed test sections, and (c) revisions only within sets of items belonging to a common stimulus (graph, table, reading passage, etc.).…”

Psychometric and Psychological Effects of Item Selection and Review on Computerized Testing

“…Typically, CATs do not incorporate item review, a fact not well received by examinees. They perceive a loss of control over the test, which increases their anxiety level and affects their performance on the test (Stocking, 1997;Wise, Freeman, Finney, Enders, & Severance, 1997).…”

Limiting Answer Review and Change on Computerized Adaptive Vocabulary Tests: Psychometric and Attitudinal Results

“…As a result, it is important that operational CATs that permit review have safeguards against score inflation that might result from use of this strategy. Stocking (1997) simulated computerized adaptive versions of the Graduate Records Examination (GRE) that incorporated several itemreview options as possible controls against achieving inflated ability estimates via the Wainer strategy. She used ML-based ability estimates that were transformed to the GRE scale score metric and the Wainer strategy as a worst-case model of examinee revision behavior and found that measurement properties were noticeably distorted.…”