Fueled by the call for formative assessments, diagnostic classification models (DCMs) have recently gained popularity in psychometrics. Despite their potential for providing diagnostic information that aids in classroom instruction and students' learning, empirical applications of DCMs to classroom assessments have been highly limited. This is partly because how DCMs with different estimation methods perform in small sample contexts is not yet wellexplored. Hence, this study aims to investigate the performance of respondent classification and item parameter estimation with a comprehensive simulation design that resembles classroom assessments using different estimation methods. The key findings are the following:(1) although the marked difference in respondent classification accuracy was not observed among the maximum likelihood (ML), Bayesian, and nonparametric methods, the Bayesian method provided slightly more accurate respondent classification in parsimonious DCMs than the ML method, and in complex DCMs, the ML method yielded the slightly better result than the Bayesian method; (2) while item parameter recovery was poor in both Bayesian and ML methods, the Bayesian method exhibited unstable slip values owing to the multimodality of their posteriors under complex DCMs, and the ML method produced irregular estimates that appear to be well-estimated due to a boundary problem under parsimonious DCMs.

No abstract

No abstract

Diagnostic classification models (DCMs) enable finer-grained inspection of the latent states of respondents' strengths and weaknesses. However, the accuracy of diagnosis deteriorates when misspecification occurs in the predefined item-attribute relationship, which is defined by a Q-matrix. To forestall misdiagnosis, several Q-matrix estimation methods have been developed in recent years; however, their scalability to large-scale assessment is extremely limited. In this study, we focus on the deterministic inputs, noisy "and" gate (DINA) model and propose a new framework for Q-matrix estimation in which the goal is to find the Q-matrix with the maximized marginal likelihood. Based on this framework, we developed a scalable estimation algorithm for the DINA Q-matrix by constructing an iteration algorithm utilizing stochastic optimization and variational inference. The simulation and empirical studies reveal that the proposed method achieves high-speed computation and good accuracy. Our method can be a useful tool for estimating a Q-matrix in large-scale settings.

No abstract