Data‐driven Q‐matrix validation using a residual‐based statistic in cognitive diagnostic assessment

Abstract

In a cognitive diagnostic assessment (CDA), attributes refer to fine-grained knowledge points or skills. The Q-matrix is a central component of CDA, which specifies the relationship between items and attributes. Oftentimes, attributes and Q-matrix are defined by subject-matter experts, and assumed to be appropriate without any misspecifications. However, this assumption does not always hold in real applications. To address this concern, this paper proposes a residual-based statistic for validating the Q-matrix. Its performance is evaluated in a simulation study and compared against that of an existing method proposed in Liu, Xu and Ying (2012, Applied Psychological Measurement, 36, 548). Simulation results indicate that the proposed method leads to a higher recovery rate of the Q-matrix and is computationally more efficient. The advantage in computational efficiency is particularly pronounced when the number of attributes measured by the test reaches five or more. Results also suggest that the two methods have different tendencies in estimating the attribute vector for each item. In cases where the methods fail to recover the correct Q-matrix, the method in Liu et al. (2012, Applied Psychological Measurement, 36, 548) tends to overestimate the number of attributes measured by the items, whereas our method does not show that bias.

Data‐driven Q‐matrix learning based on Boolean matrix factorization in cognitive diagnostic assessment

“…In addition, the calculation speed is fast. For example, using the same experimental design as Yu and Cheng (2019), when the sample size is 1,000, for a test with 30 items and six attributes, it takes only 57 s to estimate the Q‐matrix from the response data. Moreover, the existing Q‐matrix estimation methods are based on the fixed Q‐matrix design, and the performance of the method in this study is also satisfactory under the semi‐random Q‐matrix design, which shows that the algorithm is robust.…”

“…One is the fixed design (abbreviated as FD), that is, for the 20‐item test, the same Q‐matrices as in references Liu et al . (2012) and Yu and Cheng (2019) are used; for the 30‐item test, the same Q‐matrices as in references de la Torre and Chiu (2016) and Yu and Cheng (2019) are used. There are six Q‐matrices, details are all listed in Appendix A, and each satisfies sufficient conditions.…”

“…Thus, there is an urgent need for an automatic and intelligent means to construct the Q‐matrix. During the past decade, construction of a mapping relationship between items and latent attributes based on examinees' response data became a field undergoing intense study ( e.g ., Barnes, 2010; Chen, 2017; Chen et al ., 2015, 2018, 2021; Chiu, 2013; Chung, 2014; Close, 2012; de la Torre, 2008; Desmarais, 2012; Desmarais et al ., 2012; Desmarais & Naceur, 2013; Kang et al ., 2019; Lim & Drasgow, 2017; Liu, 2016; Liu et al ., 2012, 2013; Sun et al ., 2014, 2015; Wang et al ., 2020; Xiang, 2013; Xu & Shang, 2018; Yu & Cheng, 2019).…”

“…Among this research, one kind of research is named the supervised learning schema, which requires an initial Q‐matrix specified by domain experts, then validates or estimates the Q‐matrix based on the pre‐specified Q‐matrix and examinees' response data (see, e.g ., Chen, 2017; Chiu, 2013; de la Torre, 2008; Liu et al ., 2012; Yu & Cheng, 2019). Another kind of research, which does not require a pre‐specified Q‐matrix, is called the unsupervised learning schema in the literature.…”

Balancing fit and parsimony to improve Q‐matrix validation

“…Finally, in line with the pseudo-R 2 index, the residual-based approach has been previously considered for Q-matrix validation. The statistic proposed by Yu and Cheng (2019) for the reduced DINA model could be further developed for the G-DINA model and applied within the Hull method. .…”