Mutual Information Item Selection Method in Cognitive Diagnostic Computerized Adaptive Testing With Short Test Length

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) purports to combine the strengths of both CAT and cognitive diagnosis. Cognitive diagnosis models aim at classifying examinees into the correct mastery profile group so as to pinpoint the strengths and weakness of each examinee whereas CAT algorithms choose items to determine those strengths and weakness as efficiently as possible. Most of the existing CD-CAT item selection algorithms are evaluated when test length is relatively long whereas several applications of CD-CAT, such as in interim assessment, require an item selection algorithm that is able to accurately recover examinees' mastery profile with short test length. In this article, we introduce the mutual information item selection method in the context of CD-CAT and then provide a computationally easier formula to make the method more amenable in real time. Mutual information is then evaluated against common item selection methods, such as Kullback-Leibler information, posterior weighted Kullback-Leibler information, and Shannon entropy. Based on our simulations, mutual information consistently results in nearly the highest attribute and pattern recovery rate in more than half of the conditions. We conclude by discussing how the number of attributes, Q-matrix structure, correlations among the attributes, and item quality affect estimation accuracy.

Dual‐Objective Item Selection Criteria in Cognitive Diagnostic Computerized Adaptive Testing

“…The aggregate indices allow any kinds of information criteria to be weighted. For example, instead of , one may alternatively opt for the MPWKLI or the GDI index (Kaplan et al., ), Shannon entropy index (Xu et al., ), mutual information index (Wang, ) and so forth. Central interest of this study is in extensions of the KLI, and hence we consider the MPWKLI for modified versions of the ARI and ASI.…”

An Enhanced Approach to Combine Item Response Theory With Cognitive Diagnosis in Adaptive Testing

“…We address two issues innate in their original method: noncomparability of the two information addends and the arbitrary selection of weight. Notice that even though the PWKL index (Chang & Ying, 1996) is considered in (7) and in all arguments hereafter, the two new methods that will be introduced below allow any form of information criterion, including the mutual information index (Wang, 2013), or likelihood weighted KL index (Barrada, Olea, Ponsoda, & Abad, 2008), or Shannon entropy index (Xu et al, 2003), in forming the aggregate item selection index. The noncomparability appears as a result of integration.…”

Factors Affecting the Classification Accuracy and Average Length of a Variable-Length Cognitive Diagnostic Computerized Test

“…Cheng (2010) also proposed a modified version of the KL index that balanced the number of times each skill was tapped in a CD-CAT; this method was termed the maximum modified global discrimination index (MMGDI). Wang (2013) proposed mutual information (MUINF) methods of item selection for CD-CAT that were shown to lead to more accurate classifications than the PWKL in simulated testing conditions. Kaplan et al (2015) proposed a modified PWKL index (MPWKL) and the generalized DINA discrimination index.…”

“…The field of cognitive diagnostic computerized adaptive testing (CD-CAT) has seen methods proposed for selecting optimal items in diagnostic tests (Cheng, 2009(Cheng, , 2010Kaplan, de la Torre, & Barrada, 2015;Wang, 2013;Xu, Chang, & Douglas, 2003) and for stopping a variable-length CD-CAT when a reliable classification can be made (Hsu, Wang, & Chen, 2013). Similar to item response theory (IRT)-based CAT, a CD-CAT is thus expected to be more efficient than a pencil-and-paper diagnostic test: At a given stage of the test, the next item is administered based upon the examinee's performance up to that point.…”

“…Wang (2013) varied item quality and -matrix structure but in the context of fixed-length CD-CAT. Cheng (2010) focused on item selection for fixed-length tests but did not manipulate item quality or -matrix structure.…”