Theorems and Methods of a Complete Q Matrix With Attribute Hierarchies Under Restricted Q-Matrix Design

Abstract

The design of test Q matrix can directly influence the classification accuracy of a cognitive diagnostic assessment. In this paper, we focus on Q matrix design when attribute hierarchies are known prior to test development. A complete Q matrix design is proposed and theorems are presented to demonstrate that it is a necessary and sufficient condition to guarantee the identifiability of ideal response patterns. A simulation study is also conducted to detect the effects of the proposed design on a family of conjunctive diagnostic models. The results revealed that the proposed Q matrix design is the key condition for guaranteeing classification accuracy. When only one type of item pattern in R matrix is missing from the associated test Q matrix, the related attribute-wise agreement rate will decrease dramatically. When the entire R matrix is missing, both the pattern-wise and attribute-wise agreement rates will decrease sharply. This indicates that the proposed procedures for complete Q matrix design with attribute hierarchies can serve as guidelines for test blueprint development prior to item writing in a cognitive diagnostic assessment.

Complete Q-Matrices in General Attribute Structure Models

“…Characterizations of the identifiability of AHMs as well as general attribute structure models are provided in Section 6. This generalizes a recently published result on the identifiability of AHMs by Cai et al (2018), and corrects a false theorem by Köhn and Chiu (2018). While proofs are formulated within the KST framework, results are immediately recasted in CDM terms.…”

“…In other words, a Q-matrix is complete if and only if it contains the (k × k) identity matrix as a submatrix, with k the number of attributes. In recently published papers Cai et al (2018) and Köhn and Chiu (2018) independently claim that they have generalized this result to attribute hierarchies (as suggested by Leighton et al (2004)). The following formulation restates the claim within the terminology introduced above.…”

“…For introducing AHMs, and for spelling out the correspondence to KST in the subsequent section, we refer to an example taken from Cai et al (2018, Figure 2). For the terminology we refer to Leighton et al (2004).…”

“…In particular, the term attribute hierarchy is reserved for the situations considered there, where attributes are partially ordered. Cai et al (2018) assume a so-called divergent attribute hierarchy on the attributes 1, 2, 3, 4, which is illustrated in Figure 1.…”

“…Characterizations of interesting properties of CDMs (like, e.g., identifiability) were obtained at first under the assumption of independent attributes. How these results change when attributes are interdependent is currently a highly active area (e.g., Cai, Tu, & Ding, 2018;Köhn & Chiu, 2018). This research strand is known under the label Attribute Hierarchy Models (AHMs).…”