Item Review and the Rearrangement Procedure: Its process and its results

Evaluation of a New Method for Providing Full Review Opportunities in Computerized Adaptive Testing—Computerized Adaptive Testing With Salt

“…(1) We only embedded salt items in fixed item locations, although we expect the salt items’ positions should be more dynamic in practice so that examinees are less likely to identify salt items; (2) we investigated the test length of 30 items although longer tests exist in practice, and we expect the performance of CATS would be better in longer tests because it would be harder to inflate scores; (3) we randomly selected salt items from the mini form, although in practice there can be considerations of the order of presenting test items (e.g., the sequence of content strands, the order of item difficulties, etc. ); (4) the test length was fixed in this study, although many adaptive tests are administered with stopping rules that may result in variable test lengths—the inclusion of salt items will likely increase the test length while maintaining the same level of accuracy as CAT; and (5) although only CATS provides review opportunities with no restriction, a comparison of CATS with restricted CAT (e.g., Han, ; Papanastasiou, ; Stocking, ) may shed some light on providing review opportunities to examinees taking CAT. Because multistage testing (MST) allows review within each module (i.e., a group of items in one stage), a comparison between CATS and MST may be interesting too.…”

“…Because examinees feel that they have little control over the testing environments where no review is allowed, they tend to have elevated test anxiety levels which can increase the error in the examinee's ability estimation on adaptive tests (Stocking, ; Wise, ). The research in the literature has shown that, if answer changes were allowed on a test, the final estimate of an examinee's ability could be more accurate because of a reduced anxiety level and the opportunity to fix mistakes (Olea, Revuelta, Ximénez, & Abad, ; Papanastasiou, ; Wise, ). Benjamin, Cavell, and Shallenberger () reviewed 33 studies on the effects of answer changing on test performance and found that (1) 80% of examinees changed at least one answer and (2) 68% of examinees experienced a score increase due to answer changing.…”

“…Stocking () proposed three models allowing test takers to (1) change their responses at the end of the test subject to a maximum number of revisions, (2) revise their responses freely but only within each test section, or (3) revise responses only within each item set associated with a common stimulus. Papanastasiou () evaluated a rearrangement procedure that rearranged and skipped certain items with answer changes in order to better estimate the examinees’ abilities. In this method, examinees were only allowed to make a small number of answer changes.…”

Item Parameter Estimation in Multistage Designs: A Comparison of Different Estimation Approaches for the Rasch Model

“…An item revision in CATs is not possible [7,71,72], because the item selection in CATs is based on the responses already given. Hence, changing responses retrospectively may impact the measurement precision, which results in larger standard errors [69,[73][74][75][76][77][78]. Therefore, allowing item revision within CATs has been controversially discussed in the literature, even if some contributions encountered this measurement problem (see, e.g., [66,77,[79][80][81][82]).…”

The Block Item Pocket Method for Reviewable Multidimensional Computerized Adaptive Testing

“…For instance, if the maximum Fisher information method was employed, the item information for the nonoptimal item would decrease when evaluated at the new interim ability estimate. Loss of information would lead to a decrease in estimation precision, as test information is inversely proportional to the standard error of ability estimate (Papanastasiou, 2005). Furthermore, estimation precision is related to the frequency of item review.…”