Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a cutting-edge technology in educational measurement that targets at providing feedback on examinees’ strengths and weaknesses while increasing test accuracy and efficiency. To date, most CD-CAT studies have made methodological progress under simulated conditions, but little has applied CD-CAT to real educational assessment. The present study developed a Chinese reading comprehension item bank tapping into six validated reading attributes, with 195 items calibrated using data of 28,485 second to sixth graders and the item-level cognitive diagnostic models (CDMs). The measurement precision and efficiency of the reading CD-CAT system were compared and optimized in terms of crucial CD-CAT settings, including the CDMs for calibration, item selection methods, and termination rules. The study identified seven dominant reading attribute mastery profiles that stably exist across grades. These major clusters of readers and their variety with grade indicated some sort of reading developmental mechanisms that advance and deepen step by step at the primary school level. Results also suggested that compared to traditional linear tests, CD-CAT significantly improved the classification accuracy without imposing much testing burden. These findings may elucidate the multifaceted nature and possible learning paths of reading and raise the question of whether CD-CAT is applicable to other educational domains where there is a need to provide formative and fine-grained feedback but where there is a limited amount of test time.

Improving Instructional Decision‐Making Using Diagnostic Classification Models

“…That is, we do not assume any hierarchical or directional relationship among the attributes. However, as noted by Rupp et al (2010), the unstructured specification is unlikely to reflect observed data in educational contexts where hierarchical structures arise naturally (e.g., skills build on prerequisite skills)-for instance, when assessments are developed based on learning progressions, trajectories, or maps (e.g., Li et al, 2023;Yuan et al, 2022). As an example, Figure 2 shows a linear hierarchy where attribute 𝛼 𝐽 is mastered before 𝛼 𝐾 , which in turn must be mastered before 𝛼 𝐿 .…”

Introduction toJEBSSpecial Issue on Diagnostic Statistical Models

“…Chen and Wang (2023) introduce methods for inferring the underlying graphical structure of student skills to infer the nature of the hierarchical structure of skill mastery, which is a central concern for designing learning interventions. Li et al (2023) demonstrate the utility of deploying diagnostic models in computerized adaptive testing (CAT) in an application involving elementary students Chinese reading comprehension. The second collection of papers are grounded in the factor analytic tradition and provide novel extensions and applications to identify and detect beneficial educational interventions.…”