A cognitive item response theory model called the attribute hierarchy method (AHM) is introduced and illustrated. This method represents a variation of Tatsuoka's rule‐space approach. The AHM is designed explicitly to link cognitive theory and psychometric practice to facilitate the development and analyses of educational and psychological tests. The following are described: cognitive properties of the AHM; psychometric properties of the AHM, as well as a demonstration of how the AHM differs from Tatsuoka's rule‐space approach; and application of the AHM to the domain of syllogistic reasoning to illustrate how this approach can be used to evaluate the cognitive competencies required in a higher‐level thinking task. Future directions for research are also outlined.

A method for relating factors between studies based on different individuals is developed. This approach yields a measure of reltionship between all factors under consideration, a measure which may be interpreted as a correlation coefficient.

The purpose of this study is to apply the attribute hierarchy method (AHM) to a sample of SAT algebra items administered in March 2005. The AHM is a psychometric method for classifying examinees' test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance. An attribute is a description of the procedural or declarative knowledge needed to perform a task. These attributes form a hierarchy of cognitive skills that represent a cognitive model of task performance. The study was conducted in two steps. In step 1, a cognitive model was developed by having content specialists, first, review the SAT algebra items, identify their salient attributes, and order the item-based attributes into a hierarchy. Then, the cognitive model was validated by having a sample of students think aloud as they solved each item. In step 2, psychometric analyses were conducted on the SAT algebra cognitive model by evaluating the model-data fit between the expected response patterns generated by the cognitive model and the observed response patterns produced from a random sample of 5000 examinees who wrote the items. Attribute probabilities were also computed for this random sample of examinees so diagnostic inferences about their attribute-level performances could be made. We conclude the study by describing key limitations, highlighting challenges inherent to the development and analysis of cognitive diagnostic assessments, and proposing directions for future research. Note: This is a multimedia article All multimedia components are enclosed in blue. Acrobat Reader 8.0 and Acrobat Flash Player 9.0 or higher are required. These programs, which are free, can be accessed and installed from the Adobe website (www.adobe.com). This article contains "mouse-over" actions in Figure 1 to illustrate how the attributes are measured with sample algebra items from the SAT. This article also contains multimedia clips. We present the reader with three embedded videos in Figures 2 to 4 to illustrate how a student actually solves an item. Figures 5 to 10 contain embedded audio clips so the reader can hear how a student solves an item. These video and audio clips supplement our text descriptions of the attributes to provide the reader with more concrete examples about the cognitive skills that constitute each attribute and how attributes are hierarchically related.

K. Tatsuoka's rule‐space model is a statistical method for classifying examinees' test item responses into a set of attribute‐mastery patterns associated with different cognitive skills. A fundamental assumption in the model resides in the idea that test items may be described by specific cognitive skills called attributes which can include distinct procedures, skills, or processes possessed by an examinee. The rule‐space model functions by collecting and ordering information about the attributes required to solve test items and then statistically classifying examinees' test item responses into a set of attribute‐mastery patterns, each one associated with a unique cognitive blueprint. The logic of Tatsuoka's rule‐space model, as it applies to test development and analysis, is examined an this module. Controversies and unresolved issues are also presented and discussed.

The attribute hierarchy method (AHM) applied to assessment engineering is described. It is a psychometric method for classifying examinees' test item responses into a set of attribute mastery patterns associated with different components in a cognitive model of task performance. Attribute probabilities, computed using a neural network, can be estimated for each examinee thereby providing specific information about the examinee's attribute-mastery level. The pattern recognition approach described in this study relies on an explicit cognitive model to produce the expected response patterns. The expected response patterns serve as the input to the neural network. The model also yields the cognitive test specifications. These specifications identify the examinees' attribute patterns which are used as output for the neural network. The purpose of the statistical pattern recognition analysis is to estimate the probability that an examinee possess specific attribute combinations based on their observed item response patterns. Two examples using student response data from a sample of algebra items on the SAT illustrate our pattern recognition approach.