Contrasting multistage and computer-based testing: score accuracy and aberrant responding

Abstract

The goal of the present study was to compare and contrast the efficacy of a multistage testing (MST) design using three paths compared to a traditional computer-based testing (CBT) approach involving items across all ability levels. Participants were n = 627 individuals who were subjected to both a computer-based testing (CBT) instrument and a measure constructed using multistage testing to route individuals of low, middle, and high ability to content that was respective to their ability level. Comparisons between the medium of testing involved person ability accuracy estimates and evaluation of aberrant responding. The results indicated that MST assessments deviated markedly from CBT assessments, especially for low- and high-ability individuals. Test score accuracy was higher overall in MST compared to CBT, although error of measurement was enhanced for high-ability individuals during MST compared to CBT. Evaluating response patterns indicated significant amounts of Guttman-related errors during CBT compared to MST using person-fit aberrant response indicators. It was concluded that MST is associated with significant benefits compared to CBT.

Adapting the Person Fit Analysis: Ideas on Detecting Person Misfit in Computerized Adaptive Testing

“…• Lastly, investigating person fit within multistage CAT applications can offer a promising avenue for research, especially in light of recent studies such as Sideridis, Ghamdi & Zamil (2023), which compare the effectiveness of multistage CAT and traditional CAT. Their findings highlight a notable divergence in theta scores for high-ability examinees within multistage CAT frameworks, despite generally supporting multistage CAT's role in enhancing measurement accuracy.…”