Can Examinees Use Judgments of Item Difficulty to Improve Proficiency Estimates on Computerized Adaptive Vocabulary Tests?

Abstract

Recent simulation studies indicate that there are occasions when examinees can use judgments of relative item difficulty to obtain positively biased proficiency estimates on computerized adaptive tests (CATs) that permit item review and answer change. Our purpose in the study reported here was to evaluate examinees' success in using these strategies while taking CATs in a live testing setting. We taught examinees two item difficulty judgment strategies designed to increase proficiency estimates. Examinees who were taught each strategy and examinees who were taught neither strategy were assigned at random to complete vocabulary CATs under conditions in which review was allowed after completing all items and when review was allowed only within successive blocks of items. We found that proficiency estimate changes following review were significantly higher in the regular review conditions than in the strategy conditions. Failure to obtain systematically higher scores in the strategy conditions was due in large part to errors examinees made in judging the relative difficulty of CAT items.

A Closer Look at Using Judgments of Item Difficulty to Change Answers on Computerized Adaptive Tests

“…Despite these restrictions, means across conditions ranged from 7.32 (KS, KSA) to 7.88 (GKS) and showed no statistically significant differences overall ( F (6, 211) = 1.90, p = .10) with 92.6% of the participants providing ratings of either 7 or 8. These high levels of perceived adequacy of review opportunities within restricted review CAT conditions replicate findings from Vispoel et al (2000, 2002), and demonstrate once again that restricted review is viewed very favorably by most examinees.…”

“…Despite the general pattern of detrimental results from using the K and GK strategies, Vispoel et al (2002) still found that a few examinees in the K and GK conditions benefited from using those strategies. Not surprisingly, these examinees performed well in judging item difficulties and usually changed answers when the computer algorithm operated consistently.…”

“…These findings underscore that examinees will profit from the use of the K and GK strategies only when they are highly successful in judging item difficulties and the computer item selection algorithm behaves consistently. An important question left unanswered in the Vispoel et al (2002) study was whether further coaching would lead to greater success in raising proficiency estimates using the K and GK strategies. Such coaching might direct examinees to use the K and GK strategies only when they are confident in the accuracy of their item difficulty judgments and inform them that the item selection algorithm adheres less and less consistently to a correct‐harder/incorrect‐easier sequence as the test progresses.…”

“…On these tests, examinees can change answers to any items within a block, but cannot return to those items once they have gone on to the next block. In studies of CATs with review restricted within 5‐item blocks, for example, Vispoel and colleagues (Vispoel et al, 2000, 2002) found that testing time increased by only 5–11% on average with 98–100% of examinees indicating that they had adequate opportunity to review and change answers.…”

“…The mean of the item difficulty parameters ( b values) for the item pool used here was .47 (SD = 1.11); the mean for item discrimination parameters ( a values) was 1.07 (SD = .40). This item pool was used in our previous studies of reviewing items on CATs (Vispoel et al, 2000, 2002) and is described in more detail there. The items consisted of sentence fragments in which a word or phrase is capitalized.…”