Determining the Number of Attributes in Cognitive Diagnosis Modeling

Abstract

Cognitive diagnosis models (CDMs) allow classifying respondents into a set of discrete attribute profiles. The internal structure of the test is determined in a Q-matrix, whose correct specification is necessary to achieve an accurate attribute profile classification. Several empirical Q-matrix estimation and validation methods have been proposed with the aim of providing well-specified Q-matrices. However, these methods require the number of attributes to be set in advance. No systematic studies about CDMs dimensionality assessment have been conducted, which contrasts with the vast existing literature for the factor analysis framework. To address this gap, the present study evaluates the performance of several dimensionality assessment methods from the factor analysis literature in determining the number of attributes in the context of CDMs. The explored methods were parallel analysis, minimum average partial, very simple structure, DETECT, empirical Kaiser criterion, exploratory graph analysis, and a machine learning factor forest model. Additionally, a model comparison approach was considered, which consists in comparing the model-fit of empirically estimated Q-matrices. The performance of these methods was assessed by means of a comprehensive simulation study that included different generating number of attributes, item qualities, sample sizes, ratios of the number of items to attribute, correlations among the attributes, attributes thresholds, and generating CDM. Results showed that parallel analysis (with Pearson correlations and mean eigenvalue criterion), factor forest model, and model comparison (with AIC) are suitable alternatives to determine the number of attributes in CDM applications, with an overall percentage of correct estimates above 76% of the conditions. The accuracy increased to 97% when these three methods agreed on the number of attributes. In short, the present study supports the use of three methods in assessing the dimensionality of CDMs. This will allow to test the assumption of correct dimensionality present in the Q-matrix estimation and validation methods, as well as to gather evidence of validity to support the use of the scores obtained with these models. The findings of this study are illustrated using real data from an intelligence test to provide guidelines for assessing the dimensionality of CDM data in applied settings.

Research on the Cognitive Diagnosis of Chinese Listening Comprehension Ability Based on the G-DINA Model

“…Nájera et al ( 2021b ) proposed a new Q-matrix verification method, Hull's method. The research shows that this method has greater flexibility and provides a comprehensive solution to Q-matrix specification (Nájera et al, 2021a , b ). Therefore, it is necessary to explore the effectiveness of different Q-matrix modification methods in the cognitive diagnostic assessment of Chinese listening tests in the future research.…”

Cognitively Diagnostic Analysis Using the G-DINA Model in R

“…When the Q-matrix may not be entirely correct, the first step of CDM analysis should be the empirical Q-matrix evaluation, which involves validating the number of attributes and detecting the misidentified elements. To validate the number of attributes, Nájera and colleagues [27] adopted procedures for assessing the dimensionality, which were initially developed for exploratory analysis, often without a provisional Q-matrix. When the number of attributes has been validated, a host of methods have been developed for identifying misspecified elements [28][29][30][31].…”

“…Dimensionality evaluation can be conducted by the cdmTools [44] package with cdmTools::paK() and cdmTools::modelcompK() functions. The cdmTools::paK() function adopts the parallel analysis method by comparing the eigenvalues generated from principal components, Pearson correlations, and mean criterion [27,45] of the randomly resampled correlation matrices and their sample correlation matrices. The argument cor specifies the type of correlations to be used, whose default value is "both", implying using both Pearson and tetrachoric/polychoric correlations.…”

“…The cdmTools::modelcompK() function compares several model fit indices of the CDMs fitted with different Q-matrices of a specified number of attributes that are developed through the discrete factor loading method (DFL) [46] and the Hull method [47]. Nájera and colleagues [27] suggested preferring the AIC over other indices. In modelcompK() function, exploreK = 1:5 indicates that Q-matrices with one to five attributes were evaluated.…”

cdcatR: An R Package for Cognitive Diagnostic Computerized Adaptive Testing

“…It can also be verified whether the number of attributes established from theory converges with the empirical results using procedures such as parallel analysis or relative fit indicators. Researchers can use the paK() and modelcompK() functions from the cdmTools package to conduct such analyses [28]. After addressing the aspects related to the calibration of the item bank, the following sections deal with specific aspects of the CAT implementation.…”