Advances in CD-CAT: The General Nonparametric Item Selection Method

Chiu, Chia-Yi; Chang, Yuan-Pei

doi:10.1007/s11336-021-09792-z

Cited by 11 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The performance of three item selection methods (the JSD, GDI, and MPWKL) tended to be similar regardless of the calibration model used, with the JSD index being slightly better than the GDI and MPWKL methods. Among all adaptive item selection rules, the PWKL method performed the worst, which is in line with previous studies (Chang et al, 2019; Chiu & Chang, 2021).…”

Section: Resultssupporting

confidence: 91%

Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Huang²,

Liu

2023

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a cutting-edge technology in educational measurement that targets at providing feedback on examinees’ strengths and weaknesses while increasing test accuracy and efficiency. To date, most CD-CAT studies have made methodological progress under simulated conditions, but little has applied CD-CAT to real educational assessment. The present study developed a Chinese reading comprehension item bank tapping into six validated reading attributes, with 195 items calibrated using data of 28,485 second to sixth graders and the item-level cognitive diagnostic models (CDMs). The measurement precision and efficiency of the reading CD-CAT system were compared and optimized in terms of crucial CD-CAT settings, including the CDMs for calibration, item selection methods, and termination rules. The study identified seven dominant reading attribute mastery profiles that stably exist across grades. These major clusters of readers and their variety with grade indicated some sort of reading developmental mechanisms that advance and deepen step by step at the primary school level. Results also suggested that compared to traditional linear tests, CD-CAT significantly improved the classification accuracy without imposing much testing burden. These findings may elucidate the multifaceted nature and possible learning paths of reading and raise the question of whether CD-CAT is applicable to other educational domains where there is a need to provide formative and fine-grained feedback but where there is a limited amount of test time.

show abstract

Section: Resultssupporting

confidence: 91%

Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Huang²,

Liu

2023

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

show abstract

“…especially useful for distributions that cannot be quantified analytically. 1,3,5,22,[33][34][35] The fact that bootstrapping relies on simulation is its weakness, making it difficult to replicate in other studies. Furthermore, due to differences in simulation methodology, it may be difficult to compare the results of the simulation if the results of the simulation are not summarized identically.…”

Section: Discussionmentioning

confidence: 99%

“…As a result, the vast majority of packages focus upon bootstrapping and thus so too do the vast majority of studies 1,8,30‐32 . Due to the increased computational power, these nonparametric methods can be completed efficiently and are especially useful for distributions that cannot be quantified analytically 1,3,5,22,33‐35 . The fact that bootstrapping relies on simulation is its weakness, making it difficult to replicate in other studies.…”

Section: Discussionmentioning

confidence: 99%

Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically derived distributions and simulation‐based methods

Huang

2023

Health Science Reports

View full text Add to dashboard Cite

Background and Aims: All fields have seen an increase in machine-learning techniques. To accurately evaluate the efficacy of novel modeling methods, it is necessary to conduct a critical evaluation of the utilized model metrics, such as sensitivity, specificity, and area under the receiver operator characteristic curve (AUROC). For commonly used model metrics, we proposed the use of analytically derived distributions (ADDs) and compared it with simulation-based approaches.Methods: A retrospective cohort study was conducted using the England National Health Services Heart Disease Prediction Cohort. Four machine learning models (XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boost) were used. The distribution of the model metrics and covariate gain statistics were empirically derived using boot-strap simulation (N = 10,000). The ADDs were created from analytic formulas from the covariates to describe the distribution of the model metrics and compared with those of bootstrap simulation.Results: XGBoost had the most optimal model having the highest AUROC and the highest aggregate score considering six other model metrics. Based on the Anderson-Darling test, the distribution of the model metrics created from bootstrap did not significantly deviate from a normal distribution. The variance created from the ADD led to smaller SDs than those derived from bootstrap simulation, whereas the rest of the distribution remained not statistically significantly different.Conclusions: ADD allows for cross study comparison of model metrics, which is usually done with bootstrapping that rely on simulations, which cannot be replicated by the reader.

show abstract

“…等领域 (Gu & Xu, 2020, 2023Xu, 2017)。研究显示 (王立君等, 2020;Toprak, 2021;von Davier & Lee, 2019), 认知诊断在学习系统中学习者弱项诊断、报告反馈与资源推荐, 在大规模评价数据分析与细粒度诊断, 在识别问题解决策略和职业教育, 在教学干预方法或个性化补救教学效果评价等方面都发挥着重要作用。Tatsuoka (1983,1995,2009)率先提出了 Q 矩阵, 记为 ( ) Tatsuoka, 1991Tatsuoka, , 2009, 属性层级方法 (attribute hierarchy method, AHM))的期望(理想)反应模式 (Leighton et al, 2004), 确定性输入噪音与门 (deterministic inputs, noisy and gate, DINA; Haertel, 1989)和确定性输入噪音或门(deterministic inputs, noisy or gate, DINO; Templin & Henson, 2006)模型的潜在反应模式, 非参数化聚类或分类方法中理想反应模式 (康春花等 , 2017, 2023李元白等, 2018;汪文义, 丁树良等, 2015;Chiu et al, 2008Chiu et al, , 2009Chiu & Douglas, 2013;Chiu & Chang, 2021), 以及知识空间理论的问题函数 (problem function, Heller et al, 2015Heller et al, , 2017Heller, 2022) 2013; de la Torre et al., 2022;Karelitz, 2004;Ma, 2022;Sun et al, 2013;Zhan et al, 2020Zhan et al, , 2023)。 Q 矩阵设计是认知诊断测验设计中十分重要的方面(丁树良等, 2011, 2019Liu et al, 2016;Madison & Bradshaw, 2015;Tian et al, 2020;Tu et al, 2019)。设计测验各个题目所测量的属性, 即解决 Q 矩阵设计或测验蓝图问题, 是认知诊断的核心任务 (Leighton et al, 2004)。 Tatsuoka (2009)提出 et al, 1995;Liu et al, 2013;Maris, 1999;Xu, 2013;…”

unclassified