An Enhanced Approach to Combine Item Response Theory With Cognitive Diagnosis in Adaptive Testing

Abstract

Computerized adaptive testing offers the possibility of gaining information on both the overall ability and cognitive profile in a single assessment administration. Some algorithms aiming for these dual purposes have been proposed, including the shadow test approach, the dual information method (DIM), and the constraint weighted method. The current study proposed two new methods, aggregate ranked information index (ARI) and aggregate standardized information index (ASI), which appropriately addressed the noncompatibility issue inherent in the original DIM method. More flexible weighting schemes that put different emphasis on information about general ability (i.e., θ in item response theory) and information about cognitive profile (i.e., α in cognitive diagnostic modeling) were also explored. Two simulation studies were carried out to investigate the effectiveness of the new methods and weighting schemes. Results showed that the new methods with the flexible weighting schemes could produce more accurate estimation of both overall ability and cognitive profile than the original DIM. Among them, the ASI with both empirical and theoretical weights is recommended, and attribute‐level weighting scheme is preferred if some attributes are considered more important from a substantive perspective.

Dual‐Objective Item Selection Criteria in Cognitive Diagnostic Computerized Adaptive Testing

“…Later, Wang et al. () proposed transformation of the two information pieces into the ordinal scale and standardized scale. Each criterion was named as the aggregate ranked information (ARI) index and the aggregate standardized information (ASI) index.…”

“…After obtaining the response data sets, we retrofitted the generating CDM and the two‐parameter logistic model (2PLM) based on marginal maximum likelihood estimation with the expectation‐maximization algorithm (Bock & Aitkin, ; de la Torre, ) similar to Wang et al. ().…”

“…Later, Wang et al. () proposed rescaled versions of the DI index, raising the noncomparability problem of the two information addends in the DI. McGlohen and Chang () alternatively applied the shadow‐testing algorithm to simultaneously optimize the estimation of α and θ.…”

“…The dual-objective CD-CAT has been implemented based on two separate psychometric models, one from cognitive diagnostic models (CDMs; de la Torre, 2011; Haertel, 1989;Henson, Templin, & Willse, 2009;Junker & Sijtsma, 2001;Templin & Henson, 2006;von Davier, 2005) and the other from item response theory models (Birnbaum, 1968;Lord, 1980;Lord & Novick, 1968). Research suggests that the unidimensional item response model (IRM) and the DINA model (Haertel, 1989;Junker & Sijtsma, 2001) can adequately fit the same data when attributes are highly correlated or when they are linearly and hierarchically related (Wang et al, 2014). The methods evaluating the relationships among the attributes are discussed in a number of articles: see, for example, Templin, Henson, Templin, and Roussos (2008) for evaluating the correlation between the attributes and Leighton, Gierl, and Hunka (2004) and Templin and Bradshaw (2014) for testing the linear hierarchy among the attributes.…”

Testlet-Based Multidimensional Adaptive Testing

“…Makransky et al (2013), for example, showed how MAT can be used with polytomous items. Other papers have focused on employing complex model structures in MAT (Huang et al, 2012; Wang, 2014; Wang et al, 2014; Mikolajetz and Frey, in press), on MAT with exposure control (Lee et al, 2008; Diao et al, 2009; Finkelman et al, 2009), on the simultaneous consideration of multiple restrictions in MAT (Veldkamp and van der Linden, 2002; Su, 2016; Born and Frey, in press), on online calibration (Chen and Wang, 2016) and on comparing different item selection methods for composite scores and sub scores in general (Yao, 2012), with respect to different test termination criteria (Yao, 2013), and different exposure control and content managing methods (Yao, 2014). …”