Limiting Answer Review and Change on Computerized Adaptive Vocabulary Tests: Psychometric and Attitudinal Results

Abstract

Previous simulation studies of computerized adaptive tests (CATs) have revealed that the validity and precision of proficiency estimates can be maintained when review opportunities are limited to items within successive blocks. Our purpose in this study was to evaluate the effectiveness of CATs with such restricted review options in a live testing setting. Vocabulary CATs were compared under four conditions: (a) no item review allowed, (b) review allowed only within successive 5‐item blocks, (c) review allowed only within successive lO‐item blocks, and (d) review allowed only after answering all 40 items. Results revealed no trust‐worthy differences among conditions in vocabulary proficiency estimates, measurement error, or testing time. Within each review condition, ability estimates and number correct scores increased slightly after review, more answers were changed from wrong to right than from right to wrong, most examinees who changed answers improved proficiency estimates by doing so, and nearly all examinees indicated that they had an adequate opportunity to review their previous answers. These results suggest that restricting review opportunities on CATs may provide a viable way to satisfy examinee desires, maintain validity and measurement precision, and keep testing time at acceptable levels.

11 Computer-based competence tests in the national educational panel study: The challenge of mode effects

“…Similarly, the possibility of navigating within a test or booklet may interfere with individual test-taking strategies. In particular, the mode effect of a review option has been found to be mediated by student level covariates (e.g., test anxiety, Vispoel 2000) and also to be moderated by test takers' competence (e.g., Vispoel et al 2000).…”

An NCME Instructional Module on Multistage Testing

“…See Wainer (1993), Wise (1996), and Kingsbury (1996) for discussion of this and other possible score inflation strategies.) Thus, because items are chosen based on previous responses, answer changes on item‐level CATs are not permitted, despite examinees' strong desire for this opportunity (Vispoel, 1998; Vispoel, Hendrickson, & Bleiler, 2000).…”

“…A multistage test with linearly administered stages that are adaptively chosen allows examinees to preview and review items within a stage and to change their answers. Because adaptation only takes place between stages, item review and change within stages does not present difficulty for the administration algorithm or vulnerability to response strategies that allow examinees to maximize their scores (Vispoel et al, 2000).…”

Számítógép alapú adaptív és lineáris tesztek összehasonlító hatékonyságvizsgálata

“…Adaptív tesztek bevezetése során számos esetben végezték el az adaptív és a lineáris változat összehasonlító hatékonyságvizsgálatát (Al-A'Ali, 2007;Brossman és Guille, 2014;Frey, Seitz és Kröhne, 2011;Guille, Becker, Zhu, Zhang, Song és Sun, 2011;Hambleton és Xing, 2006;Jodoin, Zenisky és Hambleton, 2006;Kingsbury és Hauser, 2004;Olea, Revuelta, Ximénez és Abad, 2000;Pyper és Lilley, 2010;Rotou, Patsula, Manfred és Rizavi, 2003;Thompson és Way, 2007;Vispoel, Hendrickson és Bleiler, 2000;Zheng, 2012), az eddigi kutatások azonban főként szimulált adatbázisokkal dolgoztak. Empirikus kutatások elsősorban az egyetemista korosztály körében folytak, melyek a legtöbb esetben kis mintán végzett pilotvizsgálatok voltak.…”

“…Mivel adaptivitás csak a modulok között valósul meg, így ez nem veszélyezteti a teszt algoritmusát és segíti a tanulókat a minél magasabb pontszám elérésében (Vispoel, Hendrickson és Bleiler, 2000). Az itemalapú adaptív tesztekhez képest jóval kevesebb adminisztrációt és számítógépes számításokat igényelnek (Hendrickson, 2007;Zheng, 2012).…”