High-Efficiency Response Distribution–Based Item Selection Algorithms for Short-Length Cognitive Diagnostic Computerized Adaptive Testing

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) purports to obtain useful diagnostic information with great efficiency brought by CAT technology. Most of the existing CD-CAT item selection algorithms are evaluated when test length is fixed and relatively long, but some applications of CD-CAT, such as in interim assessment, require to obtain the cognitive pattern with a short test. The mutual information (MI) algorithm proposed by Wang is the first endeavor to accommodate this need. To reduce the computational burden, Wang provided a simplified scheme, but at the price of scale/sign change in the original index. As a result, it is very difficult to combine it with some popular constraint management methods. The current study proposes two high-efficiency algorithms, posterior-weighted cognitive diagnostic model (CDM) discrimination index (PWCDI) and posterior-weighted attribute-level CDM discrimination index (PWACDI), by modifying the CDM discrimination index. They can be considered as an extension of the Kullback-Leibler (KL) and posterior-weighted KL (PWKL) methods. A pre-calculation strategy has also been developed to address the computational issue. Simulation studies indicate that the newly developed methods can produce results comparable with or better than the MI and PWKL in both short and long tests. The other major advantage is that the computational issue has been addressed more elegantly than MI. PWCDI and PWACDI can run as fast as PWKL. More importantly, they do not suffer from the problem of scale/sign change as MI and, thus, can be used with constraint management methods together in a straightforward manner.

Attribute Discrimination Index-Based Method to Balance Attribute Coverage for Short-Length Cognitive Diagnostic Computerized Adaptive Testing

“…Cheng (2010) used KL information method to select items successively, whereas many studies have demonstrated that PWKL information method performed better than KL information method in terms of pattern and ACCR (Cheng, 2009a,b;Mao and Xin, 2013;Wang, 2013;Hsu and Wang, 2015;Zheng and Chang, 2016). And the MPWKL information method may perform even better than PWKL (Kaplan et al, 2015).…”

“…As a consequence, cognitive diagnostic tests would be conducted more frequently than traditional tests in some areas such as interim assessment (Roussos et al, 2007;Hartz and Roussos, 2008;Mao and Xin, 2013;Kaplan et al, 2015). When CD-CAT is applied to interim assessment, the AMPs of students should be obtained with shortlength tests (Zheng and Chang, 2016). Second, to the best of our knowledge, among the studies focused on short-length test, there are only two applied that CD-CAT.…”

“…The first one is practiced by Wang (2013), who introduced the mutual information (MI) item selection method in CD-CAT. And the second one is practiced by Zheng and Chang (2016), who developed two high-efficiency algorithms to select items in CD-CAT. But no study appears to have considered the situation that balances attribute coverage in the test.…”

Utilizing response times in cognitive diagnostic computerized adaptive testing under the higher‐order deterministic input, noisy ‘and’ gate model

“…However, in some situations, the opposite effects were observed for these item-selection methods. For example, when a short test was used (fewer than 10 items), the Shannon entropy method and its modified approach of the mutual information method were found to be more efficient than the PWKL method in terms of the attribute mastery classification in CD-CAT (Wang, 2013;Zheng & Chang, 2016). The PWKL method was derived based on the KL information measure to quantify the distance or discrepancy between two probability distributions.…”

A Note on the Relationship of the Shannon Entropy Procedure and the Jensen–Shannon Divergence in Cognitive Diagnostic Computerized Adaptive Testing

“…Tatsuoka & Ferguson, 2003;Xu et al, 2003), the likelihood-or posterior-weighted KL (LWKL or PWKL) index and the hybrid KL index (Cheng, 2009), the restrictive progressive or threshold PWKL index (Wang et al, 2011), the aggregate ranked information index and the aggregate standardized information index (Wang et al, 2014), the modified PWKL (MPWKL) index (Kaplan et al, 2015), the KL expected discrimination index (W. Y. Wang et al, 2015), the posterior-weighted cognitive diagnostic model (CDM) discrimination index and the posteriorweighted attribute-level CDM discrimination index (Zheng & Chang, 2016), and the information product index (Zheng et al, 2018). The second is based on the Shannon entropy, called the Shannon entropy (SHE) procedure (Cheng, 2009;McGlohen & Chang, 2008;C.…”