Comparison of Multistage Tests With Computerized Adaptive and Paper‐and‐pencil Tests

Abstract

Traditionally, the fixed‐length linear paper‐and‐pencil (P&P) mode of administration has been the standard method of test delivery. With the advancement of technology, however, the popularity of administering tests using adaptive methods like computerized adaptive testing (CAT) and multistage testing (MST) has grown in the field of measurement in both theory and practice. In practice, several standardized tests have sections that include only set‐based items. To date, there is no study in the literature that compares these testing procedures when a test is completely set‐based under various item response theory (IRT) models. This study investigates the measurement precision of MST compared to CAT and compared to P&P tests for the one‐, two‐, and three‐parameter logistic (1‐, 2‐, and 3PL) models when the test is completely set‐based. Results showed that MST performed better for the 2‐ and 3PL models than an equivalent‐length P&P test in terms of reliability and conditional standard error of measurement. In addition, findings showed that MST performed better for the 1‐ and 2PL models than for an equivalent‐length CAT test. For the 3PL model, MST and CAT performed about the same.

An overview of differential item functioning in multistage computer adaptive testing using three-parameter logistic item response theory

“…However, for classification purposes, MST proved enjoying as a good classification accuracy as that provided by CAT, with more efficient item bank usage. In a comparison of multistage tests with computerized adaptive and paperand-pencil tests, Rotou et al (2007) investigated the measurement precision of MST to CAT and paper-and-pencil tests for the three IRT models when the test is entirely set-based. The findings revealed that MST performed better in terms of reliability and conditional standard error of measurement for the 2-and 3-PL models than the same length paper-and-pencil test.…”

“…However, this efficiency is not as great as CAT administration where adaptation occurs for each test item. However, Rotou et al (2007) compared a two-stage multistage test to a CAT of the same length, both of which including only set-based items. The MSCAT had slightly higher reliability under the one-and two-parameter models and equal reliability under the three-parameter model.…”

Item Response Theory

“…Operational data from the AP ® Calculus AB exam were used as an illustration. Rotou et al (2007) compared the measurement precision, in terms of reliability and conditional standard error of measurement (CSEM), of multistage (MS), CAT, and linear tests, using 1PL, 2PL, and 3PL IRT models. They found the MS tests to be superior to CAT and linear tests for the 1PL and 2PL models, and performance of the MS and CAT to be about the same, but better than the linear for the 3PL case.…”

Számítógép alapú adaptív és lineáris tesztek összehasonlító hatékonyságvizsgálata

“…Adaptív tesztek bevezetése során számos esetben végezték el az adaptív és a lineáris változat összehasonlító hatékonyságvizsgálatát (Al-A'Ali, 2007;Brossman és Guille, 2014;Frey, Seitz és Kröhne, 2011;Guille, Becker, Zhu, Zhang, Song és Sun, 2011;Hambleton és Xing, 2006;Jodoin, Zenisky és Hambleton, 2006;Kingsbury és Hauser, 2004;Olea, Revuelta, Ximénez és Abad, 2000;Pyper és Lilley, 2010;Rotou, Patsula, Manfred és Rizavi, 2003;Thompson és Way, 2007;Vispoel, Hendrickson és Bleiler, 2000;Zheng, 2012), az eddigi kutatások azonban főként szimulált adatbázisokkal dolgoztak. Empirikus kutatások elsősorban az egyetemista korosztály körében folytak, melyek a legtöbb esetben kis mintán végzett pilotvizsgálatok voltak.…”