The Introduction and Comparability of the Computer Adaptive Gre General Test

Abstract

This report summarizes the results from two studies. The first study assessed the comparability of scores derived from linear computer-based (CBT) and computer adaptive (CAT) versions of the three GRE General Test measures. The verbal and quantitative CATs were found to produce scores that were comparable to their CBT counterparts. However, the analytical CAT produced scores that were judged not to be comparable to the analytical CBT scores. As a result, a second study was performed to further examine the analytical measure to ascertain the extent of the lack of comparability and to obtain statistics that would permit adjustments to restore comparability.Results of the additional study of the analytical measure indicated that the differences in analytical CAT and CBT scores due to the testing paradigm were large enough to require an adjustment in scores. Therefore, in order to enhance the comparability of analytical CAT and CBT scores, the analytical CAT was equated to the analytical CBT. This equating provided new analytical CAT conversions that resulted in comparable analytical CAT and CBT scores.

Can Computerized Adaptive Testing Work in Students’ Admission to Higher Education Programs in Turkey?

“…At the beginning, computers were used to deliver the same test to all the examinees in a lineer test format (Schaeffer, Steffen, Golub-Smith, Mills, & Durso, 1995). In this format, only the test delivery was different than PBT (Wang & Shin, 2010).…”

“…It is evident that in some instances, there might be problems in the ability to compare the CAT and PBT results because of differences in scoring procedures, testing conditions, and examinee characteristics (Kolen, 2000). Schaeffer et al (1995) conducted a study on GRE scores in which they revealed no comparability between scores obtained from the two testing modes. Similar results were also reported by Paek (2005), Vispoel, Rocklin, and Wang (1994), Wang, Jiao, Young, Brooks, and Olson (2008) for adaptive testing administrated in K-12 and college level testing programs.…”

Inexorable and Inevitable: The Continuing Story of Technology and Assessment

“…For those k-ı2 agencies that plan to offer the same tests on paper and computer, comparability will be a concern, especially for highstakes decisions. Although comparability has often been supported (Bridgeman, 1998;Mead & Drasgow, 1993;Schaeffer, Bridgeman, Golub-Smith, Lewis, Potenza, & Steffen, 1998;Schaeffer, Steffen, Golub-Smith, Mills, & Durso, 1995), in some instances it does not hold, as when examinees are tested in a mode different from the one in which they routinely work (Russell & Plati, 2001). Further, while it is desirable from a fairness perspective, comparability may limit innovation by preventing the computer-based version from exploiting the technology to broaden measurement beyond what traditional methods allow.…”

Test Anxiety and Test Performance: Comparing Paper‐based and Computer‐adaptive Versions of the Gre General Test

“…The ORE scores (bothmeansand variance) ofthe study samples are also reasonably closeto those ofthe ORE population. (Schaeffer et al, 1995). Reliability estimates (coefficient alpha) for the variousanxiety measures are shownin Table 2 for both paper-basedand computer-adaptive test takers.…”

“…There is some doubt, however, that test takers can judge the relative difficulty oftest itemswith muchaccuracy, thus making this situation less likely (Wise, Freeman, Finney, Enders, & Severance, 1997). In their study of the ORE GeneralTest, Schaeffer, Steffen, Golub-Smith, Mills, and Durso (1995) found that about two-thirds ofGRE CAT takers found all or most of the test questionsto be challenging (but neithertoo easy nor too difficult). Nearly one in five test takers could not tell that the CAT was a different kind of test.…”