Balancing fit and parsimony to improve Q‐matrix validation

Abstract

The Q-matrix identifies the subset of attributes measured by each item in the cognitive diagnosis modelling framework. Usually constructed by domain experts, the Q-matrix might contain some misspecifications, disrupting classification accuracy. Empirical Qmatrix validation methods such as the general discrimination index (GDI) and Wald have shown promising results in addressing this problem. However, a cut-off point is used in both methods, which might be suboptimal. To address this limitation, the Hull method is proposed and evaluated in the present study. This method aims to find the optimal balance between fit and parsimony, and it is flexible enough to be used either with a measure of item discrimination (the proportion of variance accounted for, PVAF) or a coefficient of determination (pseudo-R 2 ). Results from a simulation study showed that the Hull method consistently showed the best performance and shortest computation time, especially when used with the PVAF. The Wald method also performed very well overall, while the GDI method obtained poor results when the number of attributes was high. The absence of a cut-off point provides greater flexibility to the Hull method, and it places it as a comprehensive solution to the Q-matrix specification problem in applied settings. This proposal is illustrated using real data.

cdcatR: An R Package for Cognitive Diagnostic Computerized Adaptive Testing

“…This could be easily implemented by using the higher-order model to model the attribute relationships [54]. Fifth, in recent years there has also been very intensive research on Q-matrix validation procedures (e.g., [7,55]). The adequacy of the Q-matrix determines to some extent the quality of the classifications that can be made with the adaptive application.…”

“…This resulted in the proposal of multiple item selection rules [4][5][6]. At the same time, a great effort has been made to develop procedures to verify a fundamental input in CDM, the so-called Q-matrix, which specifies the relationships between items and attributes [7,8]. Despite these methodological advances, empirical CD-CAT applications are still scarce.…”

“…The Q-matrix can be empirically tested by employing empirical validation procedures. For example, it can be assessed whether the empirical data lead to any modification suggestions by employing the Hull method [7] implemented in the R package cdmTools [26]. The CDM and GDINA packages include other empirical validation procedures based on item discrimination [27] that could also be used for this same purpose.…”

Research on the Cognitive Diagnosis of Chinese Listening Comprehension Ability Based on the G-DINA Model

“…For example, a ϕ as low as 0.75 has been shown to perform better under demanding conditions (e.g., low-quality items; Nájera et al, 2019 ). Nájera et al ( 2021b ) proposed a new Q-matrix verification method, Hull's method. The research shows that this method has greater flexibility and provides a comprehensive solution to Q-matrix specification (Nájera et al, 2021a , b ).…”

“…Nájera et al ( 2021b ) proposed a new Q-matrix verification method, Hull's method. The research shows that this method has greater flexibility and provides a comprehensive solution to Q-matrix specification (Nájera et al, 2021a , b ). Therefore, it is necessary to explore the effectiveness of different Q-matrix modification methods in the cognitive diagnostic assessment of Chinese listening tests in the future research.…”