Assessing the Performance of Diagnostic Classification Models in Small Sample Contexts with Different Estimation Methods

Abstract

Fueled by the call for formative assessments, diagnostic classification models (DCMs) have recently gained popularity in psychometrics. Despite their potential for providing diagnostic information that aids in classroom instruction and students' learning, empirical applications of DCMs to classroom assessments have been highly limited. This is partly because how DCMs with different estimation methods perform in small sample contexts is not yet wellexplored. Hence, this study aims to investigate the performance of respondent classification and item parameter estimation with a comprehensive simulation design that resembles classroom assessments using different estimation methods. The key findings are the following:(1) although the marked difference in respondent classification accuracy was not observed among the maximum likelihood (ML), Bayesian, and nonparametric methods, the Bayesian method provided slightly more accurate respondent classification in parsimonious DCMs than the ML method, and in complex DCMs, the ML method yielded the slightly better result than the Bayesian method; (2) while item parameter recovery was poor in both Bayesian and ML methods, the Bayesian method exhibited unstable slip values owing to the multimodality of their posteriors under complex DCMs, and the ML method produced irregular estimates that appear to be well-estimated due to a boundary problem under parsimonious DCMs.

The Restricted DINA Model: A Comprehensive Cognitive Diagnostic Model for Classroom-Level Assessments

“…The likelihood-free nature of nonparametric CDMs makes them remarkably robust to those conditions that hinder parameter estimation: small sample sizes, low-quality items, and complex Q-matrices. In these situations, both the NPC and the GNPC methods outperform their parametric counterparts (i.e., DINA/ DINO, G-DINA) in terms of classification accuracy (Chiu et al, 2018;Ma et al, 2022;Oka & Okada, 2021). These results highlight the suitability of nonparametric CDMs for classroom-level educational assessments, where nonideal conditions are expected (e.g., very small sample size).…”

“…Selecting one CDM among the different possibilities (e.g., reduced or general, parametric or nonparametric) for a particular application might seem a difficult task. Accordingly, a number of studies have been recently conducted with the aim of shedding some light on this question (Chiu et al, 2018;Ma et al, 2022;Ma & Jiang, 2021;Oka & Okada, 2021;Paulsen & Svetina, 2021;Sen & Cohen, 2021;Sorrel et al, 2021). As it would be expected in any kind of statistical model, these studies consistently found that the greater the complexity of the N ájera et al…”

“…CDM, the more challenging the estimation process. Generally, parametric CDMs demand large samples (N > 500) and high-quality items (i.e., with power to discriminate between latent classes) to obtain accurate parameter estimates (Ma et al, 2022;Ma & Jiang, 2021;Oka & Okada, 2021;Sen & Cohen, 2021;Sorrel et al, 2021). This is especially true for general models, which, in addition, are greatly impacted by the complexity of the Q-matrix (i.e., the number of attributes being measured by the items), since the number of item parameters exponentially increases as the Q-matrix becomes more complex (Sorrel et al, 2021).…”

A Formal Framework for Knowledge Acquisition: Going beyond Machine Learning

“…A less restrictive truth function focuses on whether the student has learnt skill i or not, so that is the probability that the student learnt skill i at time k . The second model—the Bayesian version of Diagnostic Classification Models (DCMs) [ 57 ]—can be viewed as an extension of Illustration 1. The purpose of DCMs is not to train the student (as for knowledge tracing), but rather to diagnose the student’s (or respondent’s) current vector , where or 0 if this particular student masters skill (or attribute) i or not.…”

Improving reliability estimation in cognitive diagnosis modeling

“…General CDMs are flexible, saturated models (i.e., they estimate a success probability for all the 2 K * j reduced attribute profiles for each item) that show better model fit than reduced CDMs. However, the exponential growth of the number of item parameters as a function of the complexity of the Q-matrix might pose estimation challenges whenever the sample size is not large (Oka & Okada, 2021;Sen & Cohen, 2021). This should be a lesser problem for the reduced CDMs, such as the DINA model, which only estimate two parameters per item regardless of K * j .…”