Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees’ Cognitive Skills in Critical Reading

Abstract

The purpose of this study is to apply the attribute hierarchy method (AHM) to a subset of SAT critical reading items and illustrate how the method can be used to promote cognitive diagnostic inferences. The AHM is a psychometric procedure for classifying examinees’ test item responses into a set of attribute mastery patterns associated with different components from a cognitive model. The study was conducted in two steps. In step 1, three cognitive models were developed by reviewing selected literature in reading comprehension as well as research related to SAT Critical Reading. Then, the cognitive models were validated by having a sample of students think aloud as they solved each item. In step 2, psychometric analyses were conducted on the SAT critical reading cognitive models by evaluating the model‐data fit between the expected and observed response patterns produced from two random samples of 2,000 examinees who wrote the items. The model that provided best data‐model fit was then used to calculate attribute probabilities for 15 examinees to illustrate our diagnostic testing procedure.

Theorems and Methods of a Complete Q Matrix With Attribute Hierarchies Under Restricted Q-Matrix Design

“…There are two main differences between the above studies and our study: (1) our study focuses on cognitive diagnosis theory, whereas the above studies focused on KST; (2) our study is based on restricted Q matrix design. In many application studies (e.g., Tatsuoka, 1990 , 2009 ; Leighton et al, 2004 ; Gierl, 2007 , 2008 ; Wang and Gierl, 2011 ), Q matrix design was restricted. To address this concern, under the framework of CDMs, this study focused on identifiability based on unrestricted Q matrix design, where the test Q matrix represents a special attribute hierarchy structure (such as divergent, convergent, or linear structures) and if an item measures one attribute, then it should also measures all of its prerequisite attributes.…”

“…Hereafter, such a Q matrix is referred to as a restricted Q matrix which does not necessarily contain R matrix. Many real examples of restricted Q matrices can in diagnosis assessments from fraction subtraction (Tatsuoka, 1990 ; De La Torre, 2011 ; de la Torre et al, 2016 ;), mathematicals learning (Tatsuoka, 1990 ; Leighton and Gierl, 2007 ), critical reading (Wang and Gierl, 2011 ), syllogistic reasoning (Leighton et al, 2004 ) etc. If a diagnostic assessment is developed based on a restricted Q matrix, then a natural question is how to ensure its identity.…”

Examining DIF in the Context of CDMs When the Q-Matrix Is Misspecified

“…In a similar vein, researchers may hypothesize that some skills are related hierarchically to other skills and thus, the Q-matrix structure may result in different correlations among the attributes. Examples of several constructs, including reading comprehension and mathematics, can be found where CDMs fit to the analyses allowing for hierarchical attribute models (e.g., Gierl et al, 2008 , 2010 ; Wang and Gierl, 2011 ).…”

Developing and validating a technology-based diagnostic assessment using the evidence-centered game design approach: an example of intrapreneurship competence

“…There have been different methods used so far: (a) apply existing test specification, if a skill to task assignment already exists (Xu and von Davier 2008), (b) use eye-tracking technology which records individuals' eye movement data during stimulus processing to understand underlying solving processes (Gorin 2007), (c) invite a group of subject matter experts to comprehend underlying cognitive processes of target groups' task solution processes (Ravand 2015;Sawaki et al 2009;Sorrel et al 2016), and (d) extract skill to task assignments on the basis of target group think aloud protocols, when they solve the tasks (Gao and Rogers 2011;Jang 2009;Li and Suen 2013;Wang and Gierl 2011). Unfortunately, none of the mentioned methods are without limitations (Kunina-Habenicht et al 2009;Li and Suen 2013).…”